Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page describes how to upgrade your environment to a newAirflowversion.

About upgrade operations

In Cloud Composer 3, youdon't manage the Cloud Composer versionof your environment:

• Cloud Composer automatically upgrades the infrastructurecomponents of your environment. These components are related toCloud Composer functionality and do not change how Airflow worksor how your Airflow DAGs are executed.
• Cloud Composer doesn't automatically upgrade Airflow versionand build, Airflow components, or components closely related to Airflowworkloads. They are not changed when the infrastructure components upgradeautomatically.
• You canmanually upgrade to a new Airflowversion or build.

For example, you can use the same version and build of Airflow for severalmonths, without performing any upgrades, and your environment still receiveslatest Cloud Composer updates, fixes, and improvements of theenvironment's infrastructure components. When later you decide to move to anew version or build of Airflow, you upgrade the Airflow version in yourenvironment.

About Airflow version upgrade

Your environmentchanges the version or build of Airflow in the followingway:

• You control the Airflow version (and build) of your environment. You canperform the Airflow version upgrade operation when you choose to switchyour environment to a different version of Airflow. For example, this mighthappen if the current version and build of Airflow areno longer supported.

• Cloud Composer re-deploys your environment's Airflow componentsusing the specified Airflowversion and build.

• Cloud Composer applies Airflow configuration changes, such ascustom PyPI packages or Airflow configuration option overrides, if yourenvironment had them before the upgrade.

• Cloud Composer updates the Airflowairflow_db connection topoint to the new Cloud SQL database.

Changing the Airflow version doesn't change how you connect to the resourcesin your environment, such as the URL of your environment's bucket, orAirflow web server.

About automatic infrastructure upgrade operations

Cloud Composer periodically runsautomatic infrastructure upgrade operations:

• Automatic infrastructure upgrade operations run periodically duringmaintenance windows specified for the environment.

• It is not possible to disable automatic infrastructure upgrades inCloud Composer 3. You can control the time periods when automatic upgradescan run by specifying custom maintenance windows for your environment.

• This operation is visible in Google Cloud console andenvironment's logs, like any other long-running operation. While theoperation is running, you can't start other operations at the environmentlevel (but you can still run Airflow DAGs).

• In some cases, automatic infrastructure upgrades can restart Airflowcomponents. During such restarts, Airflow workers are gracefullyterminated with a grace period of 24 hours. If you have tasks that needmore than 24 hours to complete, consider using deferrable operators.An upgrade might result in a short period of unavailability forenvironment's infrastructure components, such as environment monitoring.

Limitations of upgrade operations

Upgrade operations have the following limitations:

• Cloud Composer releases are graduallyrolled out to all regions supported by Cloud Composer overseveral days. The latest version from a release in progress might not beavailable in your region yet.

• You can't downgrade to an earlier version or build of Airflow.

• You can't upgrade your environment if the Airflow database containsmore than 20 GBof data. During an upgrade, a warning is shown if the Airflow database sizeis more than 20GB. In this case,perform the database maintenance to reducethe database size.

• If you use the XCom mechanism to transfer files, make sure that youuse it according to Airflow's guidelines.Transferring big files or a large number of files using XCom impactsAirflow database's performance and can lead to failures when loadingsnapshots or upgrading your environment. Consider using alternatives suchas Cloud Storage to transfer large volumes of data.

Before you begin

• Make sure that youknow the differences betweenthe current version of Airflow and the version that you are upgrading to.The new version can include different versions of preinstalledpackages, patches from Cloud Composer, or changes in Airflowfunctionality, provided by the Airflow open source project. You cancheck the versions of provider packages that are included ina specific version.

• Make sure thatyour DAGs are compatible with the new Airflow version andnew versions of all provider packages (including the Airflow providerpackage for Google Cloud and Google services). Incompatible changes cancause your DAGs to no longer work.

• Unlikeautomatic infrastructure upgrades,when an environment upgrades to a new Airflow version or build, thenall running tasks are terminated. After an upgrade is completed, Airflowschedules these tasks for a retry, depending on the way you configureretries for your DAGs.

  • We recommend tocreate a new snapshot of theenvironment to be able to re-create the environment in case this is needed.

  • Your account must have a role that can trigger environment upgradeoperations. In addition, the service account of the environment must have arole that has enough permissions to perform upgrade operations. For moreinformation, seeAccess control.

  Check that your environment is up to date

  Cloud Composer displays warnings whenyour environment's Airflow build approaches its end of support date.You can use these warnings to always keep your environmentsupported.

  

  Cloud Composer keeps track of theAirflow version and buildthat your environment is based on. When it approaches theend of support date,you can see a warning in the list of environments and on theEnvironment details page.

  Note: You can always upgrade your environment, even ifthe environment's Airflow version and build are past their support period.

  To check if your environment is up to date:

  Console

  1. In Google Cloud console, go to theEnvironments page.

     Go to Environments

  2. In the list of environments, click the name of your environment.TheEnvironment details page opens.

  3. Go to theEnvironment configuration tab.

  4. In theImage version field, one of the following messages is displayed:

     • Newest available version. Your environment image is fully supported.

     • New version available. Your environment image is fully supported andyou can upgrade it to a later version.

     • Support for this image version ends in... Your environment imageapproaches the end of the full support period.

     • This version is not supported as of... Your environment is past thefull support period.

  gcloud

  This functionality is not available through Google Cloud CLI. You canView suggested upgrades instead, which shows new versionsthat are available.

  API

  This functionality is not available through API. You canView suggested upgrades instead, which shows new versionsthat are available.

  View suggested upgrades

  Cloud Composer provides a list ofAirflow buildsthat you can upgrade your environment to.

  To view Cloud Composer versions suggested for an upgrade:

  Console

  1. In Google Cloud console, go to theEnvironments page.

     Go to Environments

  2. In the list of environments, click the name of your environment.TheEnvironment details page opens.

  3. Go to theEnvironment configuration tab and clickUpgrade Image Version.

  4. For the list of suggested versions, click the Cloud ComposerImage version drop-down menu.

  gcloud

  gcloudcomposerenvironmentslist-upgrades\ENVIRONMENT_NAME\--locationLOCATION

  Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.

  Example:

  gcloudcomposerenvironmentslist-upgradesexample-environment\--locationus-central1

  API

  You can view available versions for a location. To do so, construct animageVersions.list API request.

  For example:

  // GET https://composer.googleapis.com/v1/projects/example-project/// locations/us-central1/imageVersions

  Check for PyPI package conflicts

  You can check if PyPI packages installed in your environment have anyconflicts with preinstalled packages in the newAirflow version or build.

  A successful check means that there are no conflicts in PyPI packagedependencies between the current and the specified version. However, anupgrade operation might still not be successful because of other reasons.

  Console

  To run an upgrade check for your environment:

  1. In Google Cloud console, go to theEnvironments page.

     Go to Environments

  2. In the list of environments, click the name of your environment.TheEnvironment details page opens.

  3. Go to theEnvironment configuration tab, locatetheImage version entry, and clickUpgrade.

  4. In theEnvironment version upgrade dialog, intheNew version drop-down list, selectan Airflow version or buildthat you want to upgrade to.

  5. In thePyPI packages compatibility section, clickCheck for conflicts.

  6. Wait until the check is complete. If there are PyPI package dependencyconflicts, then the displayed error messages contain details aboutconflicting packages and package versions.

  gcloud

  To run an upgrade check for your environment, run theenvironments check-upgradecommand with theAirflow version or build.that you want to upgrade to.

  gcloudcomposerenvironmentscheck-upgrade\ENVIRONMENT_NAME\--locationLOCATION\--airflow-versionVERSION

  Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.
  • VERSION withthe new Airflowversion and build that you want to upgradeto, in theairflow-x.y.z-build.t format.You can use allversion aliases.

  Example:

  gcloudcomposerenvironmentscheck-upgradeexample-environment\--locationus-central1\--airflow-versionairflow-2.10.5-build.23

  Example output:

  Waiting for [projects/example-project/locations/us-central1/environments/example-environment] to be checked for PyPI package conflicts when upgradingto composer-3-airflow-2.10.5-build.23. Operation [projects/example-project/locations/us-central1/operations/04d0e8b2-...]...done....Response:'@type': type.googleapis.com/google.cloud.orchestration.airflow.service.v1.CheckUpgradeResponsebuildLogUri: ...containsPypiModulesConflict: CONFLICTpypiConflictBuildLogExtract: |-The Cloud Build image build failed: Build failed; check build logs fordetails. Full log can be found at ...Error details: tensorboard 2.2.2 has requirementsetuptools>=41.0.0, but you have setuptools 40.3.0.

  As an alternative, you can run an upgrade check asynchronously. Use the--async argument to make an asynchronous call, then check the result withthegcloud composer operations describecommand.

  API

  Construct anenvironments.checkUpgrade APIrequest.

  Specify the image version in theimageVersion field:

  {"imageVersion":"VERSION"}

  ReplaceVERSION with the new version that you want to upgrade to, inthecomposer-3-airflow-x.y.z-build.t format.

  Check for conflicts with Cloud Composer 3

  Cloud Composer 3 is the latest major version ofCloud Composer. You can check if a Cloud Composer 2 environment canbe upgraded to Cloud Composer 3 with an upgrade check. The followingdescriptionapplies to Cloud Composer 2 environments.

  While it isn't possible to migrate your environment from Cloud Composer 2 toCloud Composer 3 in-place, you can check for compatibility problems using anupgrade check. We recommend to do it beforemigrating to Cloud Composer 3.

  To check for compatibility with Cloud Composer 3, make a regular upgrade checkof your Cloud Composer 2 environment and specify a Cloud Composer 3 versionas a target.

  Each reported conflict can have the following types:

  • BLOCKING conflicts report incompatible environment configuration elementsthat aren't supported in Cloud Composer 3. We recommend to address theseconflicts before starting a migration to Cloud Composer 3.

  • NON_BLOCKING conflicts report environment configuration elements that aredifferent in Cloud Composer 3 and might cause unexpected behavior. Forexample, changes of the default values such as default maintenance windowsare reported as non-blocking conflicts. We recommend to check if theseconflicts can cause problems before starting the migration process.

  • PyPI package conflicts are also reported as part of this check. They arereported separately from the configuration conflicts, in the same way asfor a regularPyPI package compatibility check.

  Console

  To check for compatibility with Cloud Composer 3, usegcloud CLI or the Cloud Composer API.Google Cloud console supports only checks for PyPI package compatibility.

  gcloud

  To check for compatibility with Cloud Composer 3, run theenvironments check-upgradecommand and specify a Cloud Composer 3 version as a target.

  gcloudcomposerenvironmentscheck-upgrade\ENVIRONMENT_NAME\--locationLOCATION\--image-versionCOMPOSER_3_VERSION

  Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.
  • COMPOSER_3_VERSION withthe Airflowversion and build that you want to upgradeto, in theairflow-x.y.z-build.t format. You can use allversion aliases.

  Example:

  gcloudcomposerenvironmentscheck-upgradeexample-environment\--locationus-central1\--image-versioncomposer-3-airflow-2

  Example conflict messages:

  ...- message: Error validating property [core]unit_test_mode. Overriding [core]unit_test_mode is not permitted.type: BLOCKING- message: Environment variables [GOOGLE_CLOUD_PROJECT] may not be overridden.type: BLOCKING- message: You have to specify Worker CPUs to be 0.5, 1.0 or multiples of 2.0.type: BLOCKING- message: The environment uses a default maintenance window, which isdifferent in Composer 2 and Composer 3. Unless set to a custom value, themaintenance window will be changed to a new default after the upgrade.type: NON_BLOCKING...

  As an alternative, you can run this check asynchronously. Use the--async argument to make an asynchronous call, then check the result withthegcloud composer operations describecommand.

  API

  Construct anenvironments.checkUpgrade APIrequest.

  Specify the image version in theimageVersion field:

  {"imageVersion":"COMPOSER_3_VERSION"}

  ReplaceCOMPOSER_3_VERSION with the Airflowversion and buildthat you want to upgrade to, in theairflow-x.y.z-build.t format. You canuse allversion aliases.

  Upgrade your environment

  Caution: Pause all DAGs and wait for in-progress tasks to finish beforeupgrading.

  To upgrade your environmentto a new version or build of Airflow:

  Console

  1. In Google Cloud console, go to theEnvironments page.

     Go to Environments

  2. In the list of environments, click the name of your environment.TheEnvironment details page opens.

  3. Go to theEnvironment configuration tab.

  4. Locate theImage version item and clickUpgrade.

  5. From theImage version drop-down menu, selectan Airflow version or buildthat you want to upgrade to.

  6. ClickUpgrade.

  gcloud

  gcloudcomposerenvironmentsupdate\ENVIRONMENT_NAME\--locationLOCATION\--airflow-versionVERSION

  Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.
  • VERSION withthe new Airflowversion and build that you want to upgradeto, in theairflow-x.y.z-build.t format.You can also use allversion aliases.

  For example:

  gcloudcomposerenvironmentsupdateexample-environment\--locationus-central1\--airflow-versionairflow-2.10.5-build.23

  API

  1. Construct anenvironments.patch API request.

  2. In this request:

     1. In theupdateMask parameter, specify theconfig.softwareConfig.imageVersion mask.

     2. In the request body, in theimageVersion field, specify a newversion that you want to upgrade to.

  For example:

  // PATCH https://composer.googleapis.com/v1/projects/example-project/// locations/us-central1/environments/example-environment?updateMask=// config.softwareConfig.imageVersion{"config":{"softwareConfig":{"imageVersion":"composer-3-airflow-2.10.5-build.23"}}}

  Terraform

  Theimage_version field in theconfig.software_config blockcontrolsthe Airflow version and build of your environment. In thisfield, specify a new Airflow version and build.

  resource"google_composer_environment""example"{provider=google-betaname="ENVIRONMENT_NAME"region="LOCATION"config{software_config{image_version="VERSION"}}}

  Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.the new Airflowversion and build that you want to upgradeto, in theairflow-x.y.z-build.t format.You can also use allversion aliases.

  Example:

  resource"google_composer_environment""example"{provider=google-betaname="example-environment"region="us-central1"config{software_config{image_version="airflow-2.10.5-build.23"}}}

  What's next

  • List of versions
  • Cloud Composer versioning
  • Update environments