Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page provides troubleshooting information for problems that you mightencounter while creating Cloud Composer environments.

For troubleshooting information related to updating and upgrading environments,seeTroubleshooting environment updates and upgrades.

When Cloud Composer environments are created, the majority ofissues happen because of the following reasons:

• Service account permission problems.

• Incorrect Firewall, DNS or routing information.

• Network-related issues. For example, invalid VPC configuration, IP addressconflicts, or network IP ranges that are too narrow.

• Quota-related issues.

• Incompatible Organization Policies.

Insufficient permissions to create an environment

If Cloud Composer cannot create an environment because your accounthas insufficient permissions, it outputs the following error messages:

ERROR: (gcloud.composer.environments.create) PERMISSION_DENIED: The callerdoes not have permission

or

ERROR: (gcloud.composer.environments.create) PERMISSION_DENIED: User notauthorized to act as service account <service-account-name>.The user must be granted iam.serviceAccounts.actAs permission, included inOwner, Editor, Service Account User role. See https://cloud.google.com/iam/docs/understanding-service-accounts for additional details.

Solution: Assign roles to both to your account and to the service accountof your environment as described inAccess control.

• In Cloud Composer 2, make sure thatCloud Composer Service Agentservice account(service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com)has theCloud Composer v2 API Service Agent Extension role assigned.

• Make sure thatGoogle APIs Service Agent(PROJECT_NUMBER@cloudservices.gserviceaccount.com)has theEditor role assigned.

• In the Shared VPC configuration, followConfigure Shared VPC instructions.

The service account of the environment has insufficient permissions

When creating a Cloud Composer environment, you specify a serviceaccount that runs the environment's GKE cluster nodes. If thisservice account does not have enough permissions for the requested operation,Cloud Composer outputs the following error:

Errors in: [Web server]; Error messages:  Creation of airflow web server version failed. This may be an intermittent  issue of the App Engine service. You may retry the operation later.{"ResourceType":"appengine.v1.version","ResourceErrorCode":"504","ResourceErrorMessage":"Your deployment has failed to become healthy in the allotted timeand therefore was rolled back. If you believe this was an error, try adjustingthe 'app_start_timeout_sec' setting in the 'readiness_check' section."}

Solution: Assign roles to both to your account and to the service accountof your environment as described inAccess control.

Warnings about missing IAM roles in service accounts

When an environment creation fails, Cloud Composer generates thefollowing warning message after an error occurred:The issue may be caused by missing IAM roles in the following Service Accounts....

This warning message highlights possible causes for the error.Cloud Composer checks for required roles on the service accounts inyour project, and if these roles are not present, it generates this warningmessage.

Solution: Check that service accounts mentioned in the warning message havethe required roles. For more information about roles and permissions inCloud Composer, seeAccess control.

In some cases, you can ignore this warning. Cloud Composer does notcheck individual permissions assigned to roles. For example, If you usecustom IAM roles, it is possible that the service accountmentioned in the warning message already has all required permissions. In thiscase, you can ignore this warning.

A VPC network selected for the environment does not exist

You can specify a VPC network and a subnet for your Cloud Composerenvironment when you create it. If you do not specify a VPC network, then theCloud Composer service selects thedefault VPC and thedefaultsubnet for the environment's region and zone.

If the specified VPC network and subnet do not exist, thenCloud Composer outputs the following error:

Errors in: [GKE cluster]; Error messages:        {"ResourceType":"gcp-types/container-v1:projects.locations.clusters","R        esourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"P        roject \"<your composer project>\" has no network named \"non-existing-        vpc\".","status":"INVALID_ARGUMENT","statusMessage":"Bad        Request","requestPath":"https://container.googleapis.com/        v1/projects/<your composer        project>/locations/<zone>/clusters","httpMethod":"POST"}}

Solution:

• In Cloud Composer 2, you can create environments thatuse Private Service Connect instead of VPCnetworks.
• Before creating an environment, make sure that the VPC networkand the subnet for your new environment exist.

Incorrect network configuration

Cloud Composer environment creations require proper network or DNSconfiguration. Follow these instructions to configure connectivity to GoogleAPIs and services:

• Private IP connectivity instructions

• VPC Service Controls connectivity instructions

If you configure Cloud Composer environments in a Shared VPC mode, thenalso follow theShared VPC instructions.

A Cloud Composer environment uses a subnet for cluster nodes and IP rangesfor Pods and Services. To ensure communication with these and other IP ranges,follow these instructions to configure firewall rules:

• Private IP firewall instructions

• VPC Service Controls firewall instructions

You can also check for any log entries within selectGCE Networking andSubnetwork configuration categories in Cloud Logging to see if there are anyerrors reported during environment creation:Cloud Logging

Quota issues encountered when creating environments in large-scale networks

When creating Cloud Composer environments in large-scale networks,you might encounter the following quota limitations:

• The maximum number of VPC peerings per single VPC network is reached.
• The maximum number of primary and secondary subnet IP ranges is reached.
• The maximum number of forwarding rules in the peering group for InternalTCP/UDP Load Balancing is reached.

Solution:

• In Cloud Composer 2, you can create environments thatuse Private Service Connect instead of VPCnetworks.
• In Cloud Composer 1, apply the approach recommended forCloud Composer in large-scale networks.

Incompatible organization policies

The following policies must be configured appropriately so thatCloud Composer environments can be created successfully.

Incompatible principal access boundary policies

Principal access boundary policies configured in your organizationcan be configured in a way that blocks some of your environment's operations orprevents the creation of new environments.

If this is the case, you might see the following line in the error messages:

Operations on resource are denied due to an IAM Principal Access Boundary Policy.

The components of your environment are located ina tenant and a customer project. The tenantproject is Google-managed and doesn't belong to the organization where theenvironment is located. Theservice account of your environment must havepermissions to perform operations in the tenant project.

Solution:

• Add a condition expression to the policy's binding to exclude theenvironment's service account from the policy. For an example of how toexclude a principal so that the policy doesn't apply to it, seeConditional policy bindings for principal access boundary policiesin the Identity and Access Management documentation.

Restricting services used within organization or project

Organization or project administrators can restrict what Google services can beused in their projects using thegcp.restrictServiceUsageorganization policy constraint.

When using this organization policy, it's important toallow all the services required by Cloud Composer.

400 Error Messages: Failed to deploy the Airflow web server.

This error might be caused by a failure to create a Private IP environment'sGKE cluster because of overlapping IP ranges.

Solution: Check logs for any failures in your environment's cluster andresolve the issue based on the GKE error message.

Cloud Build fails to build environment images

Applies to: Cloud Composer 2 and Cloud Composer 1.

If the Cloud Build service account(PROJECT_NUMBER@cloudbuild.gserviceaccount.com) does not have theCloud Build Service Account (roles/cloudbuild.builds.builder) role inyour project, then attempts to create or update an environment might fail withpermission-related errors.

For example, you might see thedenied: Permission "artifactregistry.repositories.uploadArtifacts" deniedmessage followed byERROR: failed to push because we ran out of retries inthe Cloud Build logs.

To solve this issue, make sure that Cloud Build service account has theCloud Build Service Account role.

What's next

• Creating environments
• Troubleshooting environment updates and upgrades
• Access control