Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

Running a business-critical application on Cloud Composer requiresmultiple parties to carry different responsibilities. While not an exhaustivelist, this document lists the responsibilities for both Google and the Customersides.

Google Responsibilities

• Hardening andpatching the Cloud Composerenvironment's components and underlying infrastructure, includingGoogle Kubernetes Engine cluster, Cloud SQL database (that hosts the Airflowdatabase), Pub/Sub, Artifact Registry and other environmentelements. In particular, this includes auto-upgrading the underlyinginfrastructure, including the GKE cluster andCloud SQL instance of an environment.

  Note: Cloud Composer 1 is in the post-maintenance mode and new versions of Cloud Composer 1 with security fixes are no longer published. Migrate to Cloud Composer 2 to get the latest version updates with security improvements.

• Protecting access to Cloud Composer environments throughincorporating access control provided by IAM,encrypting data at rest by default,providingadditional customer-managed storage encryption,encrypting data in transit.

• Providing Google Cloud integrations for Identity and Access Management, Cloud Audit Logsand Cloud Key Management Service.

• Restricting and logging Google administrative access to customers' clustersfor contractual support purposes withAccess Transparency andAccess Approval.

• Publishing information about backward incompatible changes betweenCloud Composer and Airflow versions inCloud Composer Release Notes.

• Keeping Cloud Composer documentation up to date:

  • Providing description of all functionalities provided byCloud Composer.

  • Providing troubleshooting instructions that help to keep environments ina healthy state.

  • Publishing information about known issues with workarounds (if theyexist).

• Resolving critical security incidents related to Cloud Composerenvironments and Airflow images provided by Cloud Composer(excluding customer-installed Python packages) by delivering newenvironment versions addressing the incidents.

• Depending on customer's Support Plan, troubleshooting ofCloud Composer environment health issues.

• Maintaining and expanding the functionality of theCloud Composer Terraform provider.

• Cooperating with the Apache Airflow community to maintain and developGoogle Airflow operators.

  Note: Google won't fix or troubleshoot issues in operator providersfor third-party services or products.

• Troubleshooting and, if possible, fixing issues in Airflow corefunctionalities.

Customer responsibilities

• Upgrading to new Cloud Composer and Airflow versions to keepsupport for the product and to resolve security issues onceCloud Composer service publishes a Cloud Composerversion that addresses the issues.

• Maintaining the DAGs code to keep it compatible with the used Airflow version.

• Maintaining proper permissions in IAM for the environment'sservice account. Particularly, keeping permissions required by theCloud Composer Agent and theenvironment's service account. Maintainingrequired permission for the CMEK key used for Cloud Composerenvironment encryption and rotating it according to your needs.

  Caution: We recommend toset up a user-managed service accountfor Cloud Composer environments that has only the required set ofpermissions that are necessary to run the environment and performoperations defined in your DAGs. TheComposer Worker(composer.worker) role provides this required set of permissions in mostcases. Add extra permissions to this service account only when it'snecessary for the operation of your DAGs.

• Maintaining proper permissions in IAM for the environment'sbucket.

  Caution: Users with read-write access to the following components:
  • Your environment's bucket
  • Artifact Registry repositories with container images used by:GKEPodOperator, orGKEStartPodOperator

  can deploy their own versions of DAGs or container images to an environmenteven without explicit Cloud Composer-related permissions.These DAGs or images can be later executed in your environmentwith the permissions of the Cloud Composer environmentservice account.

• Maintaining proper IAM permissions for a service accountthat performs PyPI packages installations. For more information, seeAccess control.

  Caution: Users with read-write access to the environment's bucketor those who can initiate PyPI packages installations can initiatethe process of building images on behalf of a service accountwhich is used to perform such builds. This service account is called theenvironment's service account that is specified during the environmentcreation, It can be a user-provided service account, or the defaultservice account.

• Maintaining proper end user permissions in IAM and AirflowUI Access Control configuration.

• Keeping Airflow database size below20 GB throughusing themaintenance DAG.

• Resolving all DAG parsing issues before raising support cases toCloud Customer Care.

• Naming DAGs in a proper way (for example, without using invisible characterslike SPACE or TAB in DAG names) so that metrics can be reported correctlyfor DAGs.

• Upgrade the code of DAGs so that it doesn't use deprecated operators andmigrate to their up to date alternatives. Deprecated operators might beremoved from Airflow providers, which might impact your plans to upgradeto a later Cloud Composer or Airflow version. The deprecatedoperators are also not maintained and they must be used 'as is'.

• Configuring proper IAM permissions when using secretbackends like Secret Manager so that the environment'sservice account has access to it.

• Adjusting Cloud Composer environment parameters (such as CPU andmemory for Airflow components) and Airflow configurations to meetperformance and load expectations of Cloud Composer environmentsusingCloud Composer optimization guideandenvironment scaling guide.

• Avoiding removing permissions required by Cloud Composer Agent andenvironment's service accounts (removing these permissions can lead eitherto failed management operations or to DAG and task failures).

• Keepingall services and APIs required by Cloud Composeralways enabled. These dependencies must have quotas configured at levelsrequired for Cloud Composer.

• Following recommendations and best practices forimplementing DAGs.

• Diagnosing DAG and task failures using instructions forscheduler troubleshooting,DAG troubleshooting andtriggerer troubleshooting.

• Maintaining adisaster recovery plan, including configuring and managingsnapshots to meet your data retention and business continuity needs. Google does not restore deleted environments or their database backups.

• Maintaining the Python dependency supply chain forpackages that are installed by the customer. This includestroubleshooting installation errors caused by Python dependency mismatches and defining specific version constraints when adding or modifying these packages.

What's next

• Access control with IAM
• Clean up the Airflow database
• Security overview