Create Cloud Composer environments

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page explains how to create a Cloud Composer environment.

Before you begin

Caution: It'sno longer possible to create Cloud Composer 1 environments inGoogle Cloud console. You can still create Cloud Composer 1 environmentsthrough Google Cloud CLI, Terraform, and API if your project supportscreating new Cloud Composer 1 environments.Important: For production use cases, we highly recommend to usehighly resilient environments. The Airflow databaseof your environment is zonal. During a zonal outage, Airflow tasks that areexecuted in the affected zone are interrupted and marked as failed by Airflow.Highly resilient environments are more resilient to this issue, because theirdatabases are located in two separate zones.

Step 1. Create or choose an environment's service account

When you create an environment, you specify a service account. This serviceaccount is calledenvironment's service account. Your environment uses thisservice account to perform most of the operations.

The service account for your environment is not a user account. Aservice account is a special kind of account used by an application or avirtual machine (VM) instance, not a person.

You can't change the service account of your environment later.

Warning: Your environment's service account can havetoo broad permissions on your project. Because your environment runs DAGs onbehalf of your environment's service account, users who can add and modify DAGsin your environment's bucketcan run their code on behalf of the environment's service account andexercise all permissions of this account. Make sure that you are familiarwithsecurity considerations for environment's service accountsand understand how this account interacts with permissions and roles that yougrant to individual users in your project.

If you don't have a service account for Cloud Composerenvironments in your project yet, create it.

SeeCreate environments (Terraform) for anextended example of creating a service account for your environment inTerraform.

Important: You can use the same service account for more than oneCloud Composer environment. In this case, if you grant extrapermissions to access resources in your project to this account, allenvironments that use it will get the same permissions.

To create a new service account for your environment:

  1. Create a new service account as described inthe Identity and Access Management documentation.

  2. Grant a role to it, as described in the Identity and Access Managementdocumentation. The required role isComposer Worker (composer.worker).

  3. If your environmentuses resource location restrictions,or installs PyPI packagesfrom an Artifact Registry repository orfrom a private repository, then granttheService Account User (iam.serviceAccountUser) role to theuser-managed service account that runs your environment on itself(both the principal and the resource are the same service account).

  4. To access other resources in your Google Cloud project, grantextra permissions to access those resources to this service account.TheComposer Worker (composer.worker) role provides this requiredset of permissions in most cases. Add extra permissions to this serviceaccount only when it's necessary for the operation of your DAGs.

Step 2. Basic setup

This step creates a Cloud Composer environment with defaultparameters in the specified location.

Note: All other steps in this guide explain how to customize and configuredifferent aspects of your environment. All of the remaining steps are optional.

Console

  1. In the Google Cloud console, go to theCreate environment page.

    Go to Create environment

  2. In theName field, enter a name for your environment.

    The name must start with a lowercase letter followed by up to 62 lowercaseletters, numbers, or hyphens, and can't end with a hyphen. The environmentname is used to create subcomponents for the environment, so you must providea name that is also valid as a Cloud Storagebucket name. SeeBucket naming guidelines for a listof restrictions.

  3. In theLocation drop-down list, choose a locationfor your environment.

    A location is the region where the environment is located.

  4. In theImage version drop-down list, selectaCloud Composer image with the requiredversion of Airflow.

  5. Inin theNode configuration section, in theService account drop-down list,select a service account for your environment.

    If you don't have a serviceaccount for your environment yet, seeCreate or choose an environment's service account.

gcloud

gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versionIMAGE_VERSION\--service-account"SERVICE_ACCOUNT"

Important: You must specify an image version for Cloud Composer 1. By default,Cloud Composer API uses thedefault image forCloud Composer 2.

Replace:

  • ENVIRONMENT_NAME with the name of the environment.

    The name must start with a lowercase letter followed by up to 62 lowercaseletters, numbers, or hyphens, and can't end with a hyphen. The environmentname is used to create subcomponents for the environment, so you must providea name that is also valid as a Cloud Storagebucket name. SeeBucket naming guidelines for a listof restrictions.

  • LOCATION with the region for the environment.

    A location is the region where the environment is located.

  • SERVICE_ACCOUNT with the service account for your environment.

  • IMAGE_VERSION with the name of a Cloud Composer image.

Example:

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"example-account@example-project.iam.gserviceaccount.com"

API

Construct anenvironments.create API request. Specify theconfiguration in theEnvironment resource.

{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","config":{"softwareConfig":{"imageVersion":"IMAGE_VERSION"},"nodeConfig":{"serviceAccount":"SERVICE_ACCOUNT"}}}
Important: You must specify an image version for Cloud Composer 1. By default,Cloud Composer API uses thedefault image forCloud Composer 2.

Replace:

  • PROJECT_ID with theProject ID.

  • LOCATION with the region for the environment.

    A location is the region where the environment is located.

  • ENVIRONMENT_NAME with the environment name.

    The name must start with a lowercase letter followed by up to 62 lowercaseletters, numbers, or hyphens, and can't end with a hyphen. The environmentname is used to create subcomponents for the environment, so you must providea name that is also valid as a Cloud Storagebucket name. SeeBucket naming guidelines for a listof restrictions.

  • IMAGE_VERSION with the name of a Cloud Composer image.

  • SERVICE_ACCOUNT with the service account for your environment.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","config":{"softwareConfig":{"imageVersion":"composer-1.20.12-airflow-1.10.15"},"nodeConfig":{"serviceAccount":"example-account@example-project.iam.gserviceaccount.com"}}}

Terraform

To create an environment with default parameters is a specified location,add the following resource block to your Terraform configuration and runterraform apply.

resource"google_composer_environment""example"{provider=google-betaname="ENVIRONMENT_NAME"region="LOCATION"config{software_config{image_version="IMAGE_VERSION"}node_config{service_account="SERVICE_ACCOUNT"}}}

Important: You must specify an image version for Cloud Composer 1. By default,Cloud Composer API uses thedefault image forCloud Composer 2.

Replace:

  • ENVIRONMENT_NAME with the name of the environment.

    The name must start with a lowercase letter followed by up to 62 lowercaseletters, numbers, or hyphens, and can't end with a hyphen. The environmentname is used to create subcomponents for the environment, so you must providea name that is also valid as a Cloud Storagebucket name. SeeBucket naming guidelines for a listof restrictions.

  • LOCATION with the region for the environment.

    A location is the region where the environment is located.

  • IMAGE_VERSION with the name of a Cloud Composer image.

  • SERVICE_ACCOUNT with the service account for your environment.

Example:

resource"google_composer_environment""example"{provider=google-betaname="example-environment"region="us-central1"config{software_config{image_version="composer-1.20.12-airflow-1.10.15"}node_config{service_account="example-account@example-project.iam.gserviceaccount.com"}}}

Step 3. (Optional) Configure environment scale and performance parameters

To specify the scale and performance configuration for your environment,provide the number of nodes in your environment's GKE clusterand select machine types for environment components.

Console

On theCreate environment page:

  1. In theNode configuration section:

    • Enter theNode count.

      Node count is the number of Google Kubernetes Engine nodes in the environment's cluster.By default, environments have 3 nodes.

      You can change this value after you create your environment.

    • ChooseMachine type for nodes.

      The machine type for nodes isCompute Engine machine typeused for cluster instances. This parameter determines the number of CPUs andthe amount of memory for your environment. The default machine type isn1-standard-1.

      To change this value after you create your environment, you mustmanually reconfigure the environment cluster.

    • Enter theDisk size.

      The disk size, in GB, for environment nodes. Each node in your environment hasthis amount of disk space. Select a larger disk size if you expect to store alarge volume of data in folders that are synchronized with environment VMs.For example, in the/data folder of your environment's bucket.

      The minimum size is 30 GB. The default size is 100 GB. You can'tchange this parameter after you create an environment.

    • Choose theNumber of schedulers.

      Your environment can run more than one Airflow scheduler at the sametime. Use multiple schedulers to distribute load between severalscheduler instances for better performance and reliability.

      Increasing the number of schedulers does not always improve Airflowperformance. For example, having only one scheduler might provide betterperformance than having two. This might happen when the extra scheduler is notutilized, and thus consumes resources of your environment without contributingto overall performance. The actual scheduler performance depends on thenumber of Airflow workers, the number of DAGs and tasks that run in yourenvironment, and the configuration of both Airflow and the environment.

      We recommend starting with two schedulers and then monitoring the performanceof your environment. If you change the number of schedulers, you can alwaysscale your environment back to the original number of schedulers.

      For more information about configuring multiple schedulers, seeAirflow documentation.

  2. Expand theNetworking, Airflow config overrides, and additional features item.

  3. In theCloud SQL configuration section, chooseCloud SQL machine type.

    This parameter determines the machine type for theCloud SQL instance that runs the Airflowdatabase. The default Cloud SQL machine type isdb-n1-standard-2.

  4. In theWeb server configuration section, chooseWeb server machine type.

    This parameter determines the machine type for theCompute Engine instance that runs the Airflow webserver.

    The default web server machine type iscomposer-n1-webserver-2.

gcloud

When you create an environment, following arguments control environmentscale and performance parameters:

  • --node-count specifies the number of nodes in your environment.

    Node count is the number of Google Kubernetes Engine nodes in the environment's cluster.By default, environments have 3 nodes.

    You can change this value after you create your environment.

  • --scheduler-count specifies the number of schedulers in your environment.

    Your environment can run more than one Airflow scheduler at the sametime. Use multiple schedulers to distribute load between severalscheduler instances for better performance and reliability.

    Increasing the number of schedulers does not always improve Airflowperformance. For example, having only one scheduler might provide betterperformance than having two. This might happen when the extra scheduler is notutilized, and thus consumes resources of your environment without contributingto overall performance. The actual scheduler performance depends on thenumber of Airflow workers, the number of DAGs and tasks that run in yourenvironment, and the configuration of both Airflow and the environment.

    We recommend starting with two schedulers and then monitoring the performanceof your environment. If you change the number of schedulers, you can alwaysscale your environment back to the original number of schedulers.

    For more information about configuring multiple schedulers, seeAirflow documentation.

  • --disk-size specifies the disk size for environment VMs.

    The disk size, in GB, for environment nodes. Each node in your environment hasthis amount of disk space. Select a larger disk size if you expect to store alarge volume of data in folders that are synchronized with environment VMs.For example, in the/data folder of your environment's bucket.

    The minimum size is 30 GB. The default size is 100 GB. You can'tchange this parameter after you create an environment.

  • --machine-type specifies the machine type for node VMs.

    The machine type for nodes isCompute Engine machine typeused for cluster instances. This parameter determines the number of CPUs andthe amount of memory for your environment. The default machine type isn1-standard-1.

    To change this value after you create your environment, you mustmanually reconfigure the environment cluster.

  • --cloud-sql-machine-type specifies the machine type forthe Cloud SQL instance.

    This parameter determines the machine type for theCloud SQL instance that runs the Airflowdatabase. The default Cloud SQL machine type isdb-n1-standard-2.

  • --web-server-machine-type specifies the machine type for the Airflow webserver instance.

    This parameter determines the machine type for theCompute Engine instance that runs the Airflow webserver.

    The default web server machine type iscomposer-n1-webserver-2.

gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"SERVICE_ACCOUNT"\--zoneNODE_ZONE\--node-countNODE_COUNT\--scheduler-countSCHEDULER_COUNT\--disk-sizeDISK_SIZE\--machine-typeNODE_MACHINE_TYPE\--cloud-sql-machine-typeSQL_MACHINE_TYPE\--web-server-machine-typeWS_MACHINE_TYPE

Replace:

Example:

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"example-account@example-project.iam.gserviceaccount.com"\--zoneus-central1-a\--node-count6\--scheduler-count1\--disk-size50\--machine-typen1-standard-2\--cloud-sql-machine-typedb-n1-standard-2\--web-server-machine-typecomposer-n1-webserver-2

API

When you create an environment, in theEnvironment>EnvironmentConfig resource, specifyenvironment scale and performance parameters.

{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","config":{"nodeCount":NODE_COUNT,"nodeConfig":{"machineType":"NODE_MACHINE_TYPE","diskSizeGb":DISK_SIZE,"serviceAccount":"SERVICE_ACCOUNT"},"softwareConfig":{"schedulerCount":SCHEDULER_COUNT},"databaseConfig":{"machineType":"SQL_MACHINE_TYPE"},"webServerConfig":{"machineType":"WS_MACHINE_TYPE"}}}

Replace:

  • NODE_COUNT with the number of nodes.
  • DISK_SIZE with the disk size for environment VMs, in GB.
  • NODE_MACHINE_TYPE with the machine typefor node VMs. This value must contain azone for your environment VMs.
  • SCHEDULER_COUNT with the number of schedulers.
  • SQL_MACHINE_TYPE with the machine typefor the Cloud SQL instance.
  • WS_MACHINE_TYPE with the machine typefor the Airflow web server instance.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","config":{"nodeCount":6,"nodeConfig":{"machineType":"projects/example-project/zones/us-central1-a/machineTypes/n1-standard-2","diskSizeGb":50,"serviceAccount":"example-account@example-project.iam.gserviceaccount.com"},"softwareConfig":{"schedulerCount":1},"databaseConfig":{"machineType":"db-n1-standard-2"},"webServerConfig":{"machineType":"composer-n1-webserver-2"}}}

Terraform

When you create an environment, following fields control environment scaleand performance parameters:

  • node_count in thenode_config block specifies the number of nodes inyour environment.

    Node count is the number of Google Kubernetes Engine nodes in the environment's cluster.By default, environments have 3 nodes.

    You can change this value after you create your environment.

  • disk_size_gb in thenode_config block specifies the disk size forenvironment VMs.

    The disk size, in GB, for environment nodes. Each node in your environment hasthis amount of disk space. Select a larger disk size if you expect to store alarge volume of data in folders that are synchronized with environment VMs.For example, in the/data folder of your environment's bucket.

    The minimum size is 30 GB. The default size is 100 GB. You can'tchange this parameter after you create an environment.

  • machine_type in thenode_config block specifies the machine type fornode VMs. When you specify this field, also provideaCompute Engine zone for yourenvironment VMs in thezone field.

    The machine type for nodes isCompute Engine machine typeused for cluster instances. This parameter determines the number of CPUs andthe amount of memory for your environment. The default machine type isn1-standard-1.

    To change this value after you create your environment, you mustmanually reconfigure the environment cluster.

  • machine_type in thedatabase_config block specifies the machine typefor the Cloud SQL instance.

    This parameter determines the machine type for theCloud SQL instance that runs the Airflowdatabase. The default Cloud SQL machine type isdb-n1-standard-2.

  • machine_type in theweb_server_config block specifies the machine typefor the Airflow web server instance.

    This parameter determines the machine type for theCompute Engine instance that runs the Airflow webserver.

    The default web server machine type iscomposer-n1-webserver-2.

  • scheduler_count field in thesoftware_config block specifies thenumber of schedulers in your environment. Your environment must useAirflow 2.

resource"google_composer_environment""example"{name="ENVIRONMENT_NAME"region="LOCATION"config{node_config{node_count=NODE_COUNTdisk_size_gb=DISK_SIZEmachine_type="NODE_MACHINE_TYPE"zone="NODE_ZONE"service_account="SERVICE_ACCOUNT"}software_config{scheduler_count=SCHEDULER_COUNT}database_config{machine_type="SQL_MACHINE_TYPE"}web_server_config{machine_type="WS_MACHINE_TYPE"}}}

Replace:

  • NODE_COUNT with the number of nodes.
  • DISK_SIZE with the disk size for environment VMs, in GB.
  • NODE_MACHINE_TYPE with the machine typefor node VMs.
  • NODE_ZONE with the Compute Engine zone for your environment VMs.
  • SCHEDULER_COUNT with the number of schedulers.
  • SQL_MACHINE_TYPE with the machine typefor the Cloud SQL instance.
  • WS_MACHINE_TYPE with the machine typefor the Airflow web server instance.

Example:

resource"google_composer_environment""example"{name="example-environment"region="us-central1"config{node_config{node_count=4disk_size_gb=100zone="us-central1-a"machine_type="n1-standard-2"service_account="example-account@example-project.iam.gserviceaccount.com"}software_config{scheduler_count=2}database_config{machine_type="db-n1-standard-2"}web_server_config{machine_type="composer-n1-webserver-2"}}}

Step 4. (Optional) Configure your environment's networking

Networking parameters depend on the type of environment that you want tocreate:

Additional networking options for environments are:

  • Privately used public IP addresses. If you want to use more IP addresses, your environment canprivately use certain public IP address ranges as internal, subnet IP address ranges for pods and services.
  • Authorized networks. If you want to access the control plane of your Private IP environment using HTTPS, you can useauthorized networks to specify CIDR ranges that can do so.
  • IP Masquerade agent. By using environments with the IP Masquerade agent, you can use many-to-one IP address translations in your environment's networking configurations. For more information about creating environments with IP Masquerade agent, seeEnable the IP Masquerade agent.

Console

To create a Private IP environment:

  1. Make sure that your networking is configured for the type of environment that you want to create.

  2. Expand theNetworking, Airflow config overrides, and additional features item.

  3. In theNetwork configuration section, select theEnable VPC-native (using alias-IP) checkbox.

  4. In theNetwork drop-down list, select your VPC network ID.

  5. In theSubnetwork drop-down list, select your VPC subnetwork ID.

  6. In thePod IP Address Allocation section, specify the secondary IP range for pods. You can use an existing secondary range in your VPC network, or specify a new one in the CIDR notation.

  7. In theService IP Address Allocation section, specify the secondary IP range for services. You can use an existing secondary range in your VPC network, or specify a new one in the CIDR notation.

  8. In thePrivate IP section, select theEnable private IP checkbox.

  9. In theGKE cluster master private IP section, specify an IP range for the GKE control plane:

    • To use thedefault IP range for the region where your environment is located, selectDefault IP range.

    • To specify a custom IP range, selectCustom IP range and enter a range in the CIDR notation in theGKE cluster master private IP field.

  10. Select the level access for the GKE control plane. The control plane has two endpoints. One endpoint is private, for use by cluster nodes and VMs. Another endpoint is public. You can specify the level of access for the public endpoint:

    • To enable access to the public endpoint from authorized networks, select theAccess master endpoint using its external IP address checkbox.

      Using this option sets the level of access for the control plane to"Public endpoint access enabled, authorized networks enabled". This provides restricted access to the control plane from authorized networks. By default, no source IP addresses are specified. You can addauthorized networks to the cluster.

    • To disable access to the public endpoint from authorized networks, clear theAccess master endpoint using its external IP address checkbox.

      Using this option sets the level of access for the control plane to"Public endpoint access disabled". This prevents all internet access to the control plane.

  11. In theWeb Server private IP section, specify an IP range for the Airflow web server instance.

  12. In theCloud SQL private IP section, specify an IP range forthe Cloud SQL instance.

    Caution: The172.17.0.0/16 rangeis reserved in Cloud SQL.

gcloud

Make sure that your networking is configured for the type of environment that you want to create.

When you create an environment, the following arguments controlthe networking parameters. If you omit a parameter, the default value isused.

  • --enable-private-environment enables a Private IP environment.

  • --enable-ip-alias enables VPC Native using alias IPaddresses.

    This parameter is required when using--enable-private-environment orwhen configuring secondary ranges for pods and services.

  • --network specifies your VPC network ID.

  • --subnetwork specifies your VPC subnetwork ID.

  • --cluster-secondary-range-name or--cluster-ipv4-cidr configures thethe secondary range for pods.

  • --services-secondary-range-name or--services-ipv4-cidr to configurethe secondary range for services.

  • --master-ipv4-cidr specifies a range for the GKEcontrol plane.

  • --web-server-ipv4-cidr specifies a range for the Airflow web serverinstance.

  • --cloud-sql-ipv4-cidr specifies a range for the Cloud SQLinstance.

  • --enable-private-endpoint controls the level access forthe GKE control plane. The control plane has twoendpoints. One endpoint is private, for use by cluster nodes and VMs.Another endpoint is public. You can specify the level of access for thepublic endpoint:

    • To enable access to the public endpoint from authorized networks, omit the--enable-private-endpoint argument.

      Using this option sets the level of access for the control plane to"Public endpoint access enabled, authorized networks enabled". This provides restricted access to the control plane from authorized networks. By default, no source IP addresses are specified. You can addauthorized networks to the cluster.

    • To disable access to the public endpoint from authorized networks, specify the--enable-private-endpoint argument.

      Using this option sets the level of access for the control plane to"Public endpoint access disabled". This prevents all internet access to the control plane.

  • --enable-master-authorized-networks and--master-authorized-networksarguments configureauthorized networksfor your environment.

  • --enable-privately-used-public-ips configuresprivately used public IP addresses for your environment.

  • --enable-ip-masq-agentenables the IP Masquerade agent.

Example (Private IP environment)

gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"SERVICE_ACCOUNT"\--enable-private-environment\--enable-ip-alias\--networkNETWORK_ID\--subnetworkSUBNETWORK_ID\--cluster-ipv4-cidrPODS_RANGE\--services-ipv4-cidrSERVICES_RANGE\--master-ipv4-cidrCONTROL_PLANE_RANGE\--web-server-ipv4-cidrWEB_SERVER_RANGE\--cloud-sql-ipv4-cidrSQL_RANGE

Replace:

  • NETWORK_ID with your VPC network ID.

  • SUBNETWORK_ID with your VPC subnetwork ID.

  • PODS_RANGE with the secondary range for pods.

  • SERVICES_RANGE with the secondary range for services.

  • CONTROL_PLANE_RANGE with the secondary range for the GKEcontrol plane.

  • WEB_SERVER_RANGE with the secondary range for the Airflow web serverinstance.

  • SQL_RANGE with the range for the Cloud SQL instance.

    Caution: The172.17.0.0/16 rangeis reserved in Cloud SQL.

Step 5. (Optional) Add network tags

Networktags are applied to all node VMs in your environment'scluster. Tags are used to identify valid sources or targets for networkfirewalls. Each tag within the list must comply withRFC 1035.

For example, you might want to add network tags if you plan to restricttraffic for a Private IP environment with firewall rules.

Caution: You can't change network tags later. If you want to later configurefirewall rules that target only VMs in your environment's cluster, specifynetwork tags when you create your environment.

Console

On theCreate environment page:

  1. Locate theNode configuration section.
  2. In theTags field, specify instance tags for node VMs.

gcloud

When you create an environment, following arguments control network tags:

  • --tags specifies a comma-separated list of network tags applied to allnode VMs.
gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"SERVICE_ACCOUNT"\--tagsTAGS

Replace:

  • TAGS with a comma-separated list of network tags.

Example:

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--tagsgroup1,production

API

When you create an environment, in theEnvironment>EnvironmentConfig resource, specifynetwork tags for your environment.

{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","config":{"nodeConfig":{"tags":["TAG"],"serviceAccount":"SERVICE_ACCOUNT"}}}

Replace:

  • TAG with a network tag.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","config":{"nodeConfig":{"tags":["group1","production"],"serviceAccount":"example-account@example-project.iam.gserviceaccount.com"}}}

Terraform

When you create an environment, following fields define network tags foryour environment:

  • tags field in thenode_config block specifies a comma-separated listof network tags applied to all node VMs.
resource"google_composer_environment""example"{provider=google-betaname="ENVIRONMENT_NAME"region="LOCATION"config{node_config{tags=["TAGS"]service_account="SERVICE_ACCOUNT"}}}

Replace:

  • TAGS with a comma-separated list of network tags.

Example:

resource"google_composer_environment""example"{provider=google-betaname="example-environment"region="us-central1"config{node_config{tags=["group1","production"]service_account="example-account@example-project.iam.gserviceaccount.com"}}}

Step 6. (Optional) Configure web server network access

The Airflow web server access parameters do not depend on the type of yourenvironment. Instead, you can configure web server access separately. Forexample, a Private IP environment can still have the Airflow UI accessiblefrom the internet.

You can't configure the allowed IP ranges using private IP addresses.

Console

On theCreate environment page, in theWeb server configurationsection:

  • To provide access to the Airflow web server from all IP addresses,selectAllow access from all IP addresses.

  • To restrict access only to specific IP ranges, selectAllow access only from specific IP addresses. In theIP rangefield, specify an IP range in the CIDR notation. In theDescriptionfield, specify an optional description for this range. If you want tospecify more than one range, clickAdd IP range.

  • To forbid access for all IP addresses, selectAllow access only fromspecific IP addresses and clickDelete item next to the empty rangeentry.

Note: The Airflow web server of your environment is alwaysprotected by IAM, regardless of the IP ranges that areallowed to access it.

gcloud

When you create an environment, following arguments control web serveraccess level:

Note: The Airflow web server of your environment is alwaysprotected by IAM, regardless of the IP ranges that areallowed to access it.
gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--web-server-allow-ipip_range=WS_IP_RANGE,description=WS_RANGE_DESCRIPTION

Replace:

  • WS_IP_RANGE with the IP range, in the CIDR notation, that can accessAirflow UI.
  • WS_RANGE_DESCRIPTION with the description of the IP range.

Example:

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"example-account@example-project.iam.gserviceaccount.com"\--web-server-allow-ipip_range=192.0.2.0/24,description="office net 1"\--web-server-allow-ipip_range=192.0.4.0/24,description="office net 3"

API

When you create an environment, in theEnvironment>EnvironmentConfig resource, specify web serveraccess parameters.

  • To provide access to the Airflow web server from all IP addresses,omitwebServerNetworkAccessControl.

  • To restrict access only to specific IP ranges, specify one or more rangesinallowedIpRanges.

  • To forbid access for all IP addresses, addallowedIpRanges and make it anempty list. Do not specify IP ranges in it.

Note: The Airflow web server of your environment is alwaysprotected by IAM, regardless of the IP ranges that areallowed to access it.
{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","config":{"webServerNetworkAccessControl":{"allowedIpRanges":[{"value":"WS_IP_RANGE","description":"WS_RANGE_DESCRIPTION"}]},"nodeConfig":{"serviceAccount":"SERVICE_ACCOUNT"}}}

Replace:

  • WS_IP_RANGE with the IP range, in the CIDR notation, that can accessAirflow UI.
  • WS_RANGE_DESCRIPTION with the description of the IP range.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","config":{"webServerNetworkAccessControl":{"allowedIpRanges":[{"value":"192.0.2.0/24","description":"office net 1"},{"value":"192.0.4.0/24","description":"office net 3"}]},"nodeConfig":{"serviceAccount":"example-account@example-project.iam.gserviceaccount.com"}}}

Terraform

When you create an environment, theallowed_ip_range block in theweb_server_network_access_control block contains IP ranges that can accessweb server.

Note: The Airflow web server of your environment is alwaysprotected by IAM, regardless of the IP ranges that areallowed to access it.
resource"google_composer_environment""example"{provider=google-betaname="ENVIRONMENT_NAME"region="LOCATION"config{web_server_network_access_control{allowed_ip_range{value="WS_IP_RANGE"description="WS_RANGE_DESCRIPTION"}}node_config{service_account="SERVICE_ACCOUNT"}}}

Replace:

  • WS_IP_RANGE with the IP range, in the CIDR notation, that can accessAirflow UI.
  • WS_RANGE_DESCRIPTION with the description of the IP range.

Example:

resource"google_composer_environment""example"{provider=google-betaname="example-environment"region="us-central1"config{web_server_network_access_control{allowed_ip_range{value="192.0.2.0/24"description="office net 1"},allowed_ip_range{value="192.0.4.0/24"description="office net 3"}}node_config{service_account="example-account@example-project.iam.gserviceaccount.com"}}

Step 7. (Optional) Specify Airflow configuration overrides and environment variables

You can set upAirflow configuration overrides andenvironment variables when you create an environment. As analternative, you can do it later, after your environment is created.

Some Airflow configuration optionsare blocked and youcan't override them.

For the list of available Airflow configuration options, seeConfiguration reference for Airflow 2 andAirflow 1.10.*

To specify Airflow configuration overrides and environment variables:

Console

On theCreate environment page:

  1. Expand theNetworking, Airflow config overrides, and additional features item.

  2. In theEnvironment variables section, clickAdd environment variable.

  3. Enter theName andValue for the environment variable.

  4. In theAirflow configuration overrides section, clickAdd Airflow configuration override.

  5. Enter theSection,Key, andValue for the configurationoption override.

    For example:

    SectionKeyValue
    webserverdag_orientationTB

gcloud

When you create an environment, following arguments control environmentvariables and Airflow configuration overrides:

  • --env-variables specifies a comma-separated list of environmentvariables.

    Variable names may contain upper and lowercase letters, digits, andunderscores, but they may not begin with a digit.

  • --airflow-configs specifies a comma-separated list of keys and valuesfor Airflow configuration overrides.

gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"SERVICE_ACCOUNT"\--env-variablesENV_VARS\--airflow-configsCONFIG_OVERRIDES

Replace:

  • ENV_VARS with a list of comma-separatedNAME=VALUE pairs forenvironment variables.
  • CONFIG_OVERRIDES with a list of comma-separatedSECTION-KEY=VALUEpairs for configuration overrides. Separate the name of theconfiguration section with a- symbol, followed by the key name. Forexample:core-dags_are_paused_at_creation.

Example:

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"example-account@example-project.iam.gserviceaccount.com"\--env-variablesSENDGRID_MAIL_FROM=user@example.com,SENDGRID_API_KEY=example-key\--airflow-configscore-dags_are_paused_at_creation=True,webserver-dag_orientation=TB

API

When you create an environment, in theEnvironment>EnvironmentConfig resource, specifyenvironment variables and Airflow configuration overrides.

{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","config":{"softwareConfig":{"airflowConfigOverrides":{"SECTION-KEY":"OVERRIDE_VALUE"},"envVariables":{"VAR_NAME":"VAR_VALUE",}},"nodeConfig":{"serviceAccount":"SERVICE_ACCOUNT"}}}

Replace:

  • SECTION with the section in the configuration file where the Airflowconfiguration option is located.
  • KEY with the name of the Airflow configuration option.
  • OVERRIDE_VALUE with a value of the Airflow configuration option.
  • VAR_NAME with the name of the environment variable.
  • VAR_VALUE with the value of the environment variable.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","config":{"softwareConfig":{"airflowConfigOverrides":{"core-dags_are_paused_at_creation":"True","webserver-dag_orientation":"TB"},"envVariables":{"SENDGRID_MAIL_FROM":"user@example.com","SENDGRID_API_KEY":"example-key"}},"nodeConfig":{"serviceAccount":"example-account@example-project.iam.gserviceaccount.com"}}}

Terraform

When you create an environment, following blocks control environmentvariables and Airflow configuration overrides:

  • env_variables block in thesoftware_config block specifiesenvironment variables.

    Variable names may contain upper and lowercase letters, digits, andunderscores, but they may not begin with a digit.

  • airflow_config_overrides block in thesoftware_config block specifiesAirflow configuration overrides.

resource"google_composer_environment""example"{provider=google-betaname="ENVIRONMENT_NAME"region="LOCATION"config{software_config{airflow_config_overrides={SECTION-KEY="OVERRIDE_VALUE"}env_variables={VAR_NAME="VAR_VALUE"}}node_config{service_account="SERVICE_ACCOUNT"}}}

Replace:

  • SECTION with the section in the configuration file where the Airflowconfiguration option is located.
  • KEY with the name of the Airflow configuration option.
  • OVERRIDE_VALUE with a value of the Airflow configuration option.
  • VAR_NAME with the name of the environment variable.
  • VAR_VALUE with the value of the environment variable.

Example:

resource"google_composer_environment""example"{provider=google-betaname="example-environment"region="us-central1"config{software_config{airflow_config_overrides={core-dags_are_paused_at_creation="True"webserver-dag_orientation="TB"}env_variables={SENDGRID_MAIL_FROM="user@example.com"SENDGRID_API_KEY="example-key"}}node_config{service_account="example-account@example-project.iam.gserviceaccount.com"}}}

Step 8. (Optional) Specify maintenance windows

By default, Cloud Composer 1 environments do not have definedmaintenance windows if you create them usingGoogle Cloud console, API, or Terraform. We recommend to specifymaintenance windows for your new and existing environments.

If you create your environment using gcloud CLI, your environmenthas default maintenance windows from 00:00:00 to 04:00:00 (GMT) on Friday,Saturday, and Sunday every week.

To specify custom maintenance windows for your environment:

Console

On theCreate environment page:

  1. Expand theNetworking, Airflow config overrides, and additional features item.

  2. In theMaintenance windows section, select theSet custom time for maintenance windows checkbox.

  3. In theTimezone drop-down list, choose a time zone for maintenancewindows.

  4. SetStart time,Days, andLength, so thatcombined time for the specified schedule is at least 12 hours in a 7-dayrolling window. For example, a period of 4 hours every Monday,Wednesday, and Friday provides the required amount of time.

gcloud

The following arguments define maintenance windows parameters:

  • --maintenance-window-start sets the start time of a maintenancewindow.
  • --maintenance-window-end sets the end time of a maintenance window.
  • --maintenance-window-recurrence setsthemaintenance window recurrence.
gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"SERVICE_ACCOUNT"\--maintenance-window-start'DATETIME_START'\--maintenance-window-end'DATETIME_END'\--maintenance-window-recurrence'MAINTENANCE_RECURRENCE'

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • DATETIME_START with the start date and time in thedate/time input format. Only the specified time ofthe day is used, the specified date is ignored.
  • DATETIME_END with the end date and time in thedate/time input format. Only the specified time ofthe day is used, the specified date is ignored. The specified date andtime must be after the start date.

  • MAINTENANCE_RECURRENCE withanRFC 5545 RRULE for maintenancewindows recurrence. Cloud Composer supports two formats:

  • TheFREQ=DAILY format specifies a daily recurrence.

  • TheFREQ=WEEKLY;BYDAY=SU,MO,TU,WE,TH,FR,SA format specifies arecurrence on selected days of the week.

The following example specifies a 6-hour maintenance window between 01:00 and07:00 (UTC) on Wednesdays, Saturdays, and Sundays. The 1 January, 2023date is ignored.

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"example-account@example-project.iam.gserviceaccount.com"\--maintenance-window-start'2023-01-01T01:00:00Z'\--maintenance-window-end'2023-01-01T07:00:00Z'\--maintenance-window-recurrence'FREQ=WEEKLY;BYDAY=SU,WE,SA'

API

When you create an environment, in theEnvironment>EnvironmentConfig resource, specifymaintenance windows parameters:

{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","config":{"maintenanceWindow":{"startTime":"DATETIME_START","endTime":"DATETIME_END","recurrence":"MAINTENANCE_RECURRENCE"},"nodeConfig":{"serviceAccount":"SERVICE_ACCOUNT"}}}

Replace:

  • DATETIME_START with the start date and time in thedate/time input format. Only thespecified time of the day is used, the specified date is ignored.
  • DATETIME_END with the end date and time in thedate/time input format. Only thespecified time of the day is used, the specified date is ignored. Thespecified date and time must be after the start date.

  • MAINTENANCE_RECURRENCE with an RFC 5545 RRULE for maintenance windowsrecurrence. Cloud Composer supports two formats:

  • TheFREQ=DAILY format specifies a daily recurrence.

  • TheFREQ=WEEKLY;BYDAY=SU,MO,TU,WE,TH,FR,SA format specifies arecurrence on selected days of the week.

The following example specifies a 6-hour maintenance window between 01:00 and07:00 (UTC) on Wednesdays, Saturdays, and Sundays. The 1 January, 2023date is ignored.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","config":{"maintenanceWindow":{"startTime":"2023-01-01T01:00:00Z","endTime":"2023-01-01T07:00:00Z","recurrence":"FREQ=WEEKLY;BYDAY=SU,WE,SA"},"nodeConfig":{"serviceAccount":"SERVICE_ACCOUNT"}}}

Terraform

Themaintenance_window block specifies the maintenance windows for yourenvironment:

resource"google_composer_environment""example"{provider=google-betaname="ENVIRONMENT_NAME"region="LOCATION"config{maintenance_window{start_time="DATETIME_START"end_time="DATETIME_END"recurrence="MAINTENANCE_RECURRENCE"}node_config{service_account="SERVICE_ACCOUNT"}}}

Replace:

  • DATETIME_START with the start date and time in thedate/time input format. Only thespecified time of the day is used, the specified date is ignored.
  • DATETIME_END with the end date and time in thedate/time input format. Only thespecified time of the day is used, the specified date is ignored. Thespecified date and time must be after the start date.

  • MAINTENANCE_RECURRENCE with an RFC 5545 RRULE for maintenance windowsrecurrence. Cloud Composer supports two formats:

    • TheFREQ=DAILY format specifies a daily recurrence.
    • TheFREQ=WEEKLY;BYDAY=SU,MO,TU,WE,TH,FR,SA format specifies arecurrence on selected days of the week.

The following example specifies a 6-hour maintenance window between 01:00 and07:00 (UTC) on Wednesdays, Saturdays, and Sundays. The 1 January, 2023date is ignored.

resource"google_composer_environment""example"{provider=google-betaname="example-environment"region="us-central1"config{maintenance_window{start_time="2023-01-01T01:00:00Z"end_time="2023-01-01T07:00:00Z"recurrence="FREQ=WEEKLY;BYDAY=SU,WE,SA"}}}

Step 9. (Optional) Configure data encryption (CMEK)

By default, data in your environment is encrypted with a key provided byGoogle.

To use customer-managed encryption keys (CMEK) to encrypt data in yourenvironment, follow the instructions outlined inUsing customer-managed encryption keys.

Step 10. (Optional) Specify environment labels

You can assign labels to your environments tobreak down billing costs based on these labels.

Console

On theCreate environment page:

  1. Expand theNetworking, Airflow config overrides, and additional features item.

  2. In theLabels section, clickAdd label.

  3. InKey andValue fields, specify key and value pairs for theenvironment labels.

gcloud

When you create an environment, the--labels argument specifies a comma-separated list of keys and values with environment labels.

gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"SERVICE_ACCOUNT"\--labelsLABELS

Replace:

  • LABELS with a list of comma-separatedKEY=VALUE pairs for environmentlabels.

Example:

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"example-account@example-project.iam.gserviceaccount.com"\--labelsowner=engineering-team,env=production

API

When you create an environment, in theEnvironmentresource, specify labels for your environment.

{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","labels":{"LABEL_KEY":"LABEL_VALUE"}}

Replace:

  • LABEL_KEY with a key of the environment label.
  • LABEL_VALUE with a value of the environment label.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","labels":{"owner":"engineering-team","env":"production"}}

Terraform

When you create an environment, specify labels in thelabels block (outside of theconfig block).

resource"google_composer_environment""example"{provider=google-betaname="ENVIRONMENT_NAME"region="LOCATION"labels={LABEL_KEY="LABEL_VALUE"}}

Replace:

  • LABEL_KEY with a key of the environment label.
  • LABEL_VALUE with a value of the environment label.

Example:

resource"google_composer_environment""example"{provider=google-betaname="example-environment"region="us-central1"labels={owner="engineering-team"env="production"}}

Step 11. (Optional) Configure other parameters

Other parameters include:

  • Zone for your environment nodes

    Compute Engine zone in which to deploy cluster nodes. In thisparameter, you can select a specific zone in the location of yourenvironment.

    By default, a random zone is selected automatically.

  • OAuth scopes

    OAuth scopes are a set ofGoogle API scopes to be madeavailable on all node VMs. If empty, defaults tohttps://www.googleapis.com/auth/cloud-platform.

    If you specify custom OAuth scopes, includehttps://www.googleapis.com/auth/cloud-platform in the list of specifiedscopes.

  • Python version

    If your environment uses Airflow 1.10.* and earlier Airflow versions, youcan make your environment use Python 2. The default Python version isPython 3. For more information about Python 2 support inCloud Composer, seeSupported Python versions.

Console

On theCreate environment page:

  1. In theNode configuration section:

    • In theZone drop-down list, select a zone for your environmentnodes.

      You can't change zone later.

    • In theOAuth Scopes field, specify OAuth scopes for node VMs.

      You can't change OAuth scopes later.

      To specify several OAuth scopes, provide a comma-separated list ofvalues. Includehttps://www.googleapis.com/auth/cloud-platformin the list of specified scopes.

    • In thePython version field, choose the version of Python.

      You can't change the Python version later.

gcloud

When you create an environment, following arguments control miscellaneousparameters of your environment:

  • --zone specifies a Compute Engine zone for your environmentVMs.

  • --oauth-scopes specifies a comma-separted list of OAuth scopes.Includehttps://www.googleapis.com/auth/cloud-platformin the list of specified scopes.

  • --python-version specifies the version of Python.

gcloudcomposerenvironmentscreateENVIRONMENT_NAME\--locationLOCATION\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"SERVICE_ACCOUNT"\--zoneZONE\--oauth-scopesOAUTH_SCOPES\--python-versionPYTHON_VERSION

Replace:

  • ZONE with the name of the Compute Engine zone.
  • OAUTH_SCOPES with a comma-separated list of OAuth scopes.
  • PYTHON_VERSION with the Python version (3 or2).

Example:

gcloudcomposerenvironmentscreateexample-environment\--locationus-central1\--image-versioncomposer-1.20.12-airflow-1.10.15\--service-account"example-account@example-project.iam.gserviceaccount.com"\--zoneus-central1-a\--oauth-scopeshttps://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/bigquery\--python-version3

API

When you create an environment, in theEnvironment>EnvironmentConfig resource, specifymiscellaneous parameters of your environment.

{"name":"projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME","config":{"nodeConfig":{"location":"projects/PROJECT_ID/zones/ZONE","oauthScopes":["OAUTH_SCOPE"],"serviceAccount":"SERVICE_ACCOUNT"},"softwareConfig":{"pythonVersion":"PYTHON_VERSION"}}}

Replace:

  • ZONE with the name of the Compute Engine zone.
  • OAUTH_SCOPE with an OAuth scope. To specify additional scopes, addthehttps://www.googleapis.com/auth/cloud-platform scope, followed bythe additional scope items.scopes.
  • PYTHON_VERSION with the Python version (3 or2).

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments{"name":"projects/example-project/locations/us-central1/environments/example-environment","config":{"nodeConfig":{"location":"projects/example-project/zones/us-central1-a","oauthScopes":["https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/bigquery"],"serviceAccount":"example-account@example-project.iam.gserviceaccount.com"},"softwareConfig":{"pythonVersion":"3"}}}

Terraform

When you create an environment, following fields control miscellaneousparameters of your environment:

  • zone field in thenode_config block specifies a Compute Enginezone for your environment VMs.

  • oauth_scopes field in thenode_config block specifies a comma-separatedlist of OAuth scopes.

  • python_version field in thesoftware_config block specifies theversion of Python.

resource"google_composer_environment""example"{name="ENVIRONMENT_NAME"region="LOCATION"config{node_config{zone="ZONE"oauth_scopes="[OAUTH_SCOPES]"service_account="SERVICE_ACCOUNT"}software_config{python_version="PYTHON_VERSION"}}}

Replace:

  • ZONE with the name of the Compute Engine zone.
  • OAUTH_SCOPES with a comma-separated list of OAuth scopes.
  • PYTHON_VERSION with the Python version (3 or2).

Example:

resource"google_composer_environment""example"{name="example-environment"region="us-central1"config{node_config{zone="us-central1-a"oauth_scopes="[https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/bigquery]"service_account="example-account@example-project.iam.gserviceaccount.com"}software_config{python_version="3"}}}

Step 12. (Optional) Enforce Beta API usage

Preview — Enforcing beta API usage

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

You can explicitly make your environment use beta Cloud ComposerAPI, even if your environment does not have any preview features. If you doso, your environment is created using thev1beta1 service endpoint.

Console

On theCreate environment page:

  1. Expand theNetworking, Airflow config overrides, and additional features item.

  2. In theBeta API section, select theEnforce the usage of Beta APIcheckbox.

gcloud

Create your environment usingthegcloud beta composer command.

API

Create your environment using thev1beta1 serviceendpoint.

Terraform

The Terraform provider for Cloud Composer uses beta API bydefault.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-17 UTC.