Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page is a companion to the main page aboutcreating environments. It demonstrates how to set up aCloud Composer environment and a user-managed service accountfor this environment in an existing Google Cloud project withTerraform. You can use this page as a start, then add moreconfiguration parameters for your environment, as needed.

About Terraform modules for Cloud Composer

Google maintainstheTerraform provider for Google Cloud,which includes thegoogle_composer_environment resource.This is the officially supported module for Terraform.

There are other Terraform modules maintained by the open sourcecommunity. To resolve problems with such modules, you can create issues in theirrepositories on GitHub. One example of an open source project maintaned onGitHub is theterraform-google-composer module, which is apart ofTerraform blueprints and modules for Google Cloud.While Google contributes to this project, the module is not maintained byGoogle and is not related to the Terraform provider for Google Cloud.

Before you begin

• This guide assumes that you have a Google Cloud project withconfigured billing:

  • You can use an existing project.
  • You cancreate a new projectusing Google Cloud console, Google Cloud CLI, API, or a Python client library.
  • You can create and manage your project using Terraform. For moreinformation, see Terraform documentation for thegoogle_project resource.

• Install gcloud CLI.

Authenticate with Google Cloud

To authenticate with Google Cloud, run:

gcloudauthapplication-defaultlogin

For more information about this command, seegcloud auth application-default.

Configure the Google provider in Terraform

Specify your existingproject IDand a default region for resources. Your Cloud Composerenvironment uses this region.

Thegoogle-beta provider supports Cloud Composer featuresthat are in Preview. If you want to use only GA features ofCloud Composer, use thegoogle provider instead ofgoogle-beta.

provider"google-beta"{project="example-project"region="us-central1"}

Enable the Cloud Composer API

Enable the Cloud Composer API in your project:

resource"google_project_service""composer_api"{provider=google-betaproject="example-project"service="composer.googleapis.com"  // Disabling Cloud Composer API might irreversibly break all other  // environments in your project.  // This parameter prevents automatic disabling  // of the API when the resource is destroyed.  // We recommend to disable the API only after all environments are deleted.disable_on_destroy=false  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will  //prevent you from disabling composer_api through Terraform if any environment was  //there in the last 30 dayscheck_if_service_has_usage_on_destroy=true}

Create an environment's service account in your project

This guide demonstrates how to create an environment's service account that hasall required permissions to run a Cloud Composer environment.

We strongly recommend toset up a user-managed service accountfor your Cloud Composer environments that has onlypermissions required to run your environment and operations in your DAGs,as described in this guide.

The service account of your environment might need additional permissions toaccess other resources in your project. For example, if your DAGs transfer datainto BigQuery, this account might need permissions or rolesspecific to BigQuery.

Define a custom service account with the following roles and permissions:

resource"google_service_account""custom_service_account"{provider=google-betaaccount_id="custom-service-account"display_name="Example Custom Service Account"}resource"google_project_iam_member""custom_service_account"{provider=google-betaproject="example-project"member=format("serviceAccount:%s",google_service_account.custom_service_account.email)  // Role for Public IP environmentsrole="roles/composer.worker"}

Create an environment

Create your environment using Terraform.

The example demonstrates how to create an environment that uses a customservice account. You can add more parameters that define other configurationparameters of your environment, such as custom scale and performanceparameters, or additional PyPI packages.

For more information about other parameters, seeCreate environments.

resource"google_composer_environment""example_environment"{provider=google-betaname="example-environment"config{software_config{image_version="composer-3-airflow-2.10.5-build.23"}node_config{service_account=google_service_account.custom_service_account.email}}}

Full Terraform script

provider"google-beta"{project="example-project"region="us-central1"}resource"google_project_service""composer_api"{provider=google-betaproject="example-project"service="composer.googleapis.com"  // Disabling Cloud Composer API might irreversibly break all other  // environments in your project.disable_on_destroy=false  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will  //prevent you from disabling composer_api through Terraform if any environment was  //there in the last 30 dayscheck_if_service_has_usage_on_destroy=true}resource"google_service_account""custom_service_account"{provider=google-betaaccount_id="custom-service-account"display_name="Example Custom Service Account"}resource"google_project_iam_member""custom_service_account"{provider=google-betaproject="example-project"member=format("serviceAccount:%s",google_service_account.custom_service_account.email)  // Role for Public IP environmentsrole="roles/composer.worker"}resource"google_composer_environment""example_environment"{provider=google-betaname="example-environment"config{software_config{image_version="composer-3-airflow-2.10.5-build.23"}node_config{service_account=google_service_account.custom_service_account.email}}}

What's next

See other documentation pages for information about configuring yourenvironment with Terraform. For example:

• Create environments
• Override Airflow configuration options
• Set environment variables
• Install Python dependencies
• Scale environments