Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page describes how to install Python packages for yourCloud Composer environment.

About packages in Cloud Composer

This section explains how PyPI packages work in Cloud Composer.

Preinstalled and custom PyPI packages in Cloud Composer images

Cloud Composer images contains both preinstalled and custom PyPIpackages.

• Preinstalled PyPI packages are packages that are included inthe Cloud Composer image of your environment. EachCloud Composer image containsPyPI packages that are specific for your version ofCloud Composer and Airflow.

• Custom PyPI packages are packages that you can install in your environmentin addition to preinstalled packages.

Options to manage PyPI packages for Cloud Composer environments

Before you begin

• You must have a role that can trigger environment updateoperations. In addition, the service account of the environment must havea role that has enough permissions to perform updateoperations.

  Important: When you install PyPI packages in your environment,Airflow executes Python code from the installed package. Make sure thatyou install packages that come from sources that you can trust.Users with thecomposer.environments.update permission can execute Pythoncode on behalf of the environment's service account when installingpackages in your environment. For more information, seeAccess control.

• If your environment is protected by a VPC Service Controls perimeter,then before installing PyPI dependencies you mustgrant additional user identities with access to services thatthe service perimeter protects and enable support for a private PyPIrepository.

• Requirements must follow the format specifiedinPEP-508 where each requirement is specified inlowercase and consists of the package name with optional extras and versionspecifiers.

• If a dependency conflict causes the update to fail, your environmentcontinues running with its existing dependencies. If the operationsucceeds, you can begin using the newly installed Python dependencies inyour DAGs.

• PythonVirtualenvOperator doesn't use the configuration from your environment'spip.conf file. If you want to use specific dependencies, includinginstalling them from a custom index, pass requirements in therequirementsparameter as a list of strings. For more information about the format, seeRequirements File Format in the pipdocumentation.

View the list of PyPI packages

You can get the list of packages for your environment in several formats.

View preinstalled packages

To view the list of preinstalled packages for your environment, seethe list of packages for theCloud Composer image of your environment.

View all packages

To view all packages (both preinstalled and custom) in your environment:

gcloudbetacomposerenvironmentslist-packages\ENVIRONMENT_NAME\--locationLOCATION

View custom PyPI packages

gcloudcomposerenvironmentsdescribeENVIRONMENT_NAME\--locationLOCATION\--format="value(config.softwareConfig.pypiPackages)"

Install custom packages in a Cloud Composer environment

This section describes different methods for installing custom packages in yourenvironment.

Install packages from PyPI

A package can be installed fromPython Package Index if it has no externaldependencies or conflicts with preinstalled packages.

scipy>=0.13.3scikit-learnnltk[machine_learning]

gcloudcomposerenvironmentsupdateENVIRONMENT_NAME\--locationLOCATION\--update-pypi-packages-from-filerequirements.txt

gcloudcomposerenvironmentsupdateENVIRONMENT_NAME\--locationLOCATION\--update-pypi-packagePACKAGE_NAMEEXTRAS_AND_VERSION

gcloudcomposerenvironmentsupdateexample-environment\--locationus-central1\--update-pypi-package"scipy>=0.13.3"

gcloudcomposerenvironmentsupdateENVIRONMENT_NAME\--locationLOCATION\--remove-pypi-packagesPACKAGE_NAMES

gcloudcomposerenvironmentsupdateexample-environment\--locationus-central1\--remove-pypi-packagesscipy,scikit-learn

// PATCH https://composer.googleapis.com/v1/projects/example-project/// locations/us-central1/environments/example-environment?updateMask=// config.softwareConfig.pypiPackages.EXAMPLE_PACKAGE,// config.softwareConfig.pypiPackages.ANOTHER_PACKAGE{"config":{"softwareConfig":{"pypiPackages":{"EXAMPLE_PACKAGE":"","ANOTHER_PACKAGE":">=1.10.3"}}}}

resource"google_composer_environment""example"{name="ENVIRONMENT_NAME"region="LOCATION"config{software_config{pypi_packages={PACKAGE_NAME="EXTRAS_AND_VERSION"}}}}

resource"google_composer_environment""example"{name="example-environment"region="us-central1"config{software_config{pypi_packages={scipy=">=1.10.3"scikit-learn=""nltk="[machine_learning]"}}}}

Install packages from a public repository

You can install packages hosted in other repositories that have a public IPaddress.

The packages must beproperly configured, so that thedefaultpip tool can install it.

1. Create apip.conf file and include the followinginformation in the file, if applicable:

   • URL of the repository (in theindex-url parameter)
   • Access credentials for the repository
   • Non-defaultpip installation options

   Example:

   [global]index-url=https://example.com/

2. (Optional) In some cases, you might want to fetch packages from multiplerepositories, such as when the public repository contains some specificpackages that you want to install, and you want to install all otherpackages from PyPI:

   1. Configure anArtifact Registry virtual repository.
   2. Add configuration for multiple repositories (including PyPI, if needed) and define the order in whichpip searches the repositories.
   3. Specify the virtual repository's URL in theindex-url parameter.

3. Determine the URI of your environment's bucket.

4. Upload the pip.conf file to the/config/pip/folder in your environment's bucket.

5. Install packages usingone of the available methods.

Install packages from an Artifact Registry repository

You canstore packages in an Artifact Registry repositoryin your project, and configure your environment to install from it.

Configure roles and permissions:

1. Make sure that the Cloud Build service account haspermissions to read from your Artifact Registry repository.

2. If your environment has restricted access to other services in yourproject, for example, if you use VPC Service Controls:

   1. Assign permissions to access your Artifact Registry repository tothe environment's service account instead of theCloud Build service account.

   2. Make sure that connectivity to the Artifact Registry repository isconfigured in your project.

To install custom PyPI packages from an Artifact Registry repository:

1. Create apip.conf file and include the following information in the file, if applicable:

   • URL of the Artifact Registry repository (in theindex-url parameter)
   • Access credentials for the repository
   • Non-defaultpip installation options

   For an Artifact Registry repository, append/simple/ to the repositoryURL:

   [global]index-url = https://us-central1-python.pkg.dev/example-project/example-repository/simple/

2. (Optional) In some cases, you might want to fetch packages from multiplerepositories, such as when your Artifact Registry repository containssome specific packages that you want to install, and you want to installall other packages from PyPI:

   1. Configure anArtifact Registry virtual repository.
   2. Add configuration for multiple repositories (including PyPI, if needed) and define the order in whichpip searches the repositories.
   3. Specify the virtual repository's URL in theindex-url parameter.

3. Upload this pip.conf file to the/config/pip/folder in your environment's bucket. For example:gs://us-central1-example-bucket/config/pip/pip.conf.

4. Install packages usingone of the available methods.

Install packages from a private repository

You can host a private repository in your project's network and configure yourenvironment to install Python packages from it.

Configure roles and permissions:

1. If you install custom PyPI packages from a repository in your project'snetwork, and this repository does not have a public IP address:

   1. Assign permissions to access this repository to the environment'sservice account.

   2. Make sure that connectivity to this repository is configured in yourproject.

To install packages from a private repository hosted in your project's network:

1. Create apip.conf file and include the following information in the file, if applicable:

   • IP address of the repository in your project's network
   • Access credentials for the repository
   • Non-defaultpip installation options

   Example:

   [global]index-url=https://192.0.2.10/

2. (Optional) In some cases, you might want to fetch packages from multiplerepositories, such as when the private repository contains some specificpackages that you want to install, and you want to install all otherpackages from PyPI:

   1. Configure anArtifact Registry virtual repository.
   2. Add configuration for multiple repositories (including PyPI, if needed) and define the order in whichpip searches the repositories.
   3. Specify the virtual repository's URL in theindex-url parameter.

3. (Optional)Youcan use a custom certificate when installing packages from your privaterepository. To do so:

   1. Upload the certificate file to the/config/pip/ folder in your environment's bucket.

      Caution:Don't upload private keys. Upload only a publiccertificate bundle in the PEM format. Verify that the certificatebundle contains the correct root CA.

   2. In pip.conf, specify the name of the certificate file in thecertparameter. Don't change the/etc/pip/ folder.

      Example:

      [global]cert =/etc/pip/example-certificate.pem

4. Upload the pip.conf file to the/config/pip/folder in your environment's bucket. For example:gs://us-central1-example-bucket/config/pip/pip.conf.

5. Install packages usingone of the available methods.

Install a local Python library

To install an in-house or local Python library:

1. Place the dependencies within a subdirectory in thedags/ folder in your environment's bucket. To import a module from asubdirectory, each subdirectory in the module's path must containan__init__.py package marker file.

   In the following example, the dependency iscoin_module.py:

   dags/  use_local_deps.py  # A DAG file.  dependencies/    __init__.py    coin_module.py

2. Import the dependency from the DAG definition file.

   For example:

fromdependenciesimportcoin_module

• Use theKubernetesPodOperator. Set the Operator image to a custombuild image. If you experience packages that fail during installation dueto an unmet system dependency, use this option.

• Upload the shared object libraries to your environment's bucket. If your PyPIpackages have installed successfully but fail at runtime, use this option.

  1. Manually find the shared object libraries for the PyPI dependency(an .so file).
  2. Upload the shared object libraries to the/plugins folder in yourenvironment's bucket.
  3. Set the following environment variable:LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/airflow/gcs/plugins

Install packages in private IP environments

Depending on how you configure your project, your environment might not haveaccess to the public internet.

By default, your Cloud Composer 3 environmentcan access the internet wheninstalling PyPI packages. This works in all environment's networkingtypes (Private and Public IP). For example, you can install packages fromPython Package Index in a Private IP environment.

For more information, seeConfigure access to the internet when installing PyPI packages.

Private IP environment with public internet access

If your private IP environment can access public internet, then you caninstall packages using options for public IP environments:

• Install from PyPI. In this case, no special configuration is required.Follow the procedure described inInstall a package from PyPI.
• Install from a repository with a public IP address.Follow the procedure described inInstall a package from a private repository.
• Install from aprivate PyPI repository hosted in yourproject's network.

Private IP environment without internet access

If your private IP environment does not have access to public internet, thenyou can install packages using one of the following ways:

• Use aprivate PyPI repository hosted in your project'snetwork.
• Use aproxy server VM in your project's network to connectto a PyPI repository on the public internet. Specify the proxy address inthe/config/pip/pip.conf file in your environment's bucket.
• Use an Artifact Registry repository as the only sourceof packages. To do so, redefine theindex-url parameter, as described.
• If your security policy permits access to external IP addresses from yourVPC network, you can enable the installation of packages from repositorieson the public internet by configuringCloud NAT.
• Put Python dependencies into the/dags folder in your environment's buckettoinstall them as local libraries. This might not be agood option if the dependency tree is large.

Install to a private IP environment under resource location restrictions

Keeping your project in line with Resource Location Restrictionrequirements prohibits the use of some tools. In particular, Cloud Buildcannot be used for package installation, preventing direct access torepositories on the public internet.

To install Python dependencies in such an environment, follow the guidance fora private IP environmentswithout internet access.

Install a Python dependency to a private IP environment in a VPC Service Controls perimeter

Protecting your project with aVPC Service Controls perimeterresults in further security restrictions. In particular, Cloud Buildcannot be used for package installation, preventing direct access torepositories on the public internet.

To install Python dependencies for a private IP environment inside a perimeter,follow the guidance for private IP environmentswithout internet access.

What's next

• Troubleshooting package installation
• Troubleshooting updates and upgrades