Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page explains how to migrate DAGs, data and configuration from yourexisting Cloud Composer 2 environment to a new Cloud Composer 3 environmentusing themigration script.

Other migration guides

About the migration script

The migration script is a Python script for side-by-side migrations thatautomates the migration process from Cloud Composer 2 to Cloud Composer 3. Itusesenvironment snapshots to transfer the environment'sconfiguration to the new environment.

The script performs the following actions:

1. Obtains the configuration of the Cloud Composer 2 environment.

2. Creates a Cloud Composer 3 environment with configuration that matches theobtained configuration.

   Because Cloud Composer 3 environments have adifferent architecture, some parameters might be adjusted tomatch the differences. You can also adjust most of the environment'sparameters later.

3. Pauses all DAGs in the Cloud Composer 2 environment one by one. only dagsthat were unpaused in the Cloud Composer 2 environment will be unpausedlater.

   Note: if your environment has a large number of DAGs, this process can takea long time. You can use thecomposer_dags scriptto pause all DAGs before performing the migration and to unpause the DAGsafter the migration finishes. In this case, the migration script won'tunpause the DAGs. For more information, see theREADME file of the composer_dags script on GitHub.

4. Saves a snapshot of the source Cloud Composer 2 environment. The snapshotis saved in the default location for snapshots, the Cloud Composer 2environment's bucket.

5. Loads the snapshot to the Cloud Composer 3 environment.

   The script doesn't check the compatibility of custom PyPI packages,environment variables, and Airflow configuration option overrides withthe Cloud Composer 3 environment.

   In case of conflicts, the migration fails after the Cloud Composer 3environment is created, during the process of loading the snapshot. Inthis case, you can either adjust the configuration of yourCloud Composer 2 environment to resolve the conflict, ormigrate without the migration scriptand skip loading custom PyPI packages, environment variables, orAirflow configuration overrides when you load the snapshot.

6. Unpauses the DAGs in the Cloud Composer 3 environment. If some DAGs werealready paused before you ran the script, they will remain paused.

The script has the following limitations:

• The script always creates a new Cloud Composer 3 environment. It's notpossible to load the snapshot to an existing Cloud Composer 3 environment.To do so, you canmigrate using snapshots, without usingthe migration script.

• The script creates a Cloud Composer 3 environment only in the same region andproject as the Cloud Composer 2 environment.

• You can only load snapshots to the same or later version ofAirflow. For example, you can't load a snapshot from Airflow 2.10.2 toAirflow 2.9.3.

• Only Cloud Composer 2 environments can be migrated with the migrationscript.

Before you begin

• Because the migration script creates an environment, and then saves and loadsa snapshot, the migration process can take over one hour of time.

• The script usessnapshots. Snapshots are supported

• in Cloud Composer 2 version 2.0.9 and later versions.

• Your account requiresan IAM role that cancreate environments, save snapshots, and load snapshots.

• The maximum size of the Airflow database that supports snapshots is20 GB. If your environment's database takesmore than 20 GB,reduce the size of the Airflow database.

• The total number of objects in the/dags,/plugins and/data folders inthe environment's bucket must be less than 100,000 tocreate snapshots.

• If you use the XCom mechanism to transfer files, make sure that youuse it according to Airflow's guidelines.Transferring big files or a large number of files using XCom impactsAirflow database's performance and can lead to failures when loadingsnapshots or upgrading your environment. Consider using alternatives suchas Cloud Storage to transfer large volumes of data.

Migrate to Cloud Composer 3

This section describes the migration process using the migration script.

Check the differences between Cloud Composer 2 and Cloud Composer 3

Check the list of differencesbetween Cloud Composer 2 and Cloud Composer 3.

Make sure that your environment doesn't use features that aren't yetavailable in Cloud Composer 3 and that you are familiar with how to use andconfigure features specific to Cloud Composer 3.

Make sure that your DAGs are compatible with Cloud Composer 3

Make sure that your DAGs are compatible with Cloud Composer 3 by followingthese suggestions:

• Thelist of packages in the Cloud Composer 3 environmentcan be different than in your Cloud Composer 2 environment.This might affect the compatibility of your DAGs with Cloud Composer 3.

• In Cloud Composer 3, the environment's clusteris located in the tenant project. Make sure that your DAGs arecompatible with this change. In particular,KubernetesPodOperator workloads now scale independently fromyour environment and it's not possible to use Pod affinity configs.

Check for configuration compatibility

You cando an upgrade check to see if yourCloud Composer 2 environment's configuration is compatible withCloud Composer 3. We recommend to resolve all blocking conflicts that arereported by this check before starting the migration.

Install script's dependencies

• The script requires Python version 3.8 and later versions.

• The migration script uses gcloud CLI andcurl utilities. Makesure that both utilities are installed on your computer.

Download the script

Download the migration script (composer_migrate.py) from itsrepository on GitHub.

Authorize in gcloud CLI

Authorize in gcloud CLI:

gcloudauthlogin

Preview the new environment's parameters

You can preview the Cloud Composer 3 environment's parameters beforemigrating. You can use this to see how the Cloud Composer 2 environment'sconfiguration corresponds to Cloud Composer 3.

Airflow configuration option overrides, custom PyPI packages, and environmentvariables are loaded from the environment's snapshot and are not displayed inthe preview.

python3composer_migrate.py\--projectPROJECT_ID\--locationLOCATION\--source_environmentCOMPOSER_2_ENV\--target_environmentCOMPOSER_3_ENV\--target_airflow_versionCOMPOSER_3_AIRFLOW_VERSION\--dry_run

python3composer_migrate.py\--projectexample-project\--locationus-central1\--source_environmentexample-composer-2-environment\--target_environmentexample-composer-3-environment\--target_airflow_version2.10.2

Check environment's health

Make sure that your Cloud Composer 2 environment that you want to migrate ishealthy.

If your environment isn't healthy, the migration process will fail aftercreating a new Cloud Composer 3 environment because it won't be possible tocreate a snapshot.

SeeUse the monitoring dashboard for moreinformation about ways to check environment's health and database health.

Run the migration script

python3composer_migrate.py\--projectPROJECT_ID\--locationLOCATION\--source_environmentCOMPOSER_2_ENV\--target_environmentCOMPOSER_3_ENV\--target_airflow_versionCOMPOSER_3_AIRFLOW_VERSION

• PROJECT_ID: theProject ID.
• COMPOSER_2_ENV: the name of your Cloud Composer 2 environment.
• LOCATION: the region where the Cloud Composer 2 environment is located.The Cloud Composer 3 environment will be created in the same region.
• COMPOSER_3_AIRFLOW_VERSION: the version of Airflow of the Cloud Composer 3environment. This version must be the same or later version than in theCloud Composer 2 environment and must be one of theversionsavailable in Cloud Composer 3.

Check for DAG errors

1. Inthe Airflow web interface,go toDAGs and check for reported DAG syntax errors.

2. Check that DAG runs are scheduled at the correct time.

3. Wait for the DAG runs to happen in the Cloud Composer 3 environment andcheck if they were successful. If a DAG run was successful, don'tunpause it in the Cloud Composer 2 environment; if you do so, a DAGrun for the same time and date happens in your Cloud Composer 2environment.

4. If a specific DAG runs fails, attempt totroubleshoot the DAG until it successfullyruns in Cloud Composer 3.

Monitor your Cloud Composer 3 environment

Monitor your Cloud Composer 3 environment for potential issues, failed DAGruns, and overall environment health.

If the Cloud Composer 3 environment runs without problemsfor a sufficient period of time, consider deleting the Cloud Composer 2environment.

What's next

• Troubleshooting DAGs
• Troubleshooting environment creation