Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page explains how to configure SMTP services for yourCloud Composer environment.

Before you begin

• If your Cloud Composer environment isconfigured for Private IP, make sure that it hasconnectivity to the API endpoint for your external email service (such ashttps://api.sendgrid.com, or your preferred SMTP server).

Configure SendGrid email services

To receive notifications, configure your environment variablesto send email through the SendGrid email service.

Sign up with SendGrid

If you haven't already, sign up with SendGrid in Google Cloud console, clickManage On Provider to go to the SendGrid domain, and click Settings to retrieveyour username and to create an API key. As a Google Cloud developer, you canstart with 12,000 free emails per month.

Go to SendGrid Email API

Configure variables

SendGrid uses the API key and a From email address for sending mail. Youcan provide this information with one of the following options:

• (Recommended)Store values in Secret Manager

  Note: This option is available in Airflow 2.1.4 and later versions. InAirflow versions from 2.1.4 to 2.2.3, you still need to use an environmentvariable to set the email address, but you can store the SendGrid API key ina secret.

• Store values in Airflow

  Caution: If you use this option, the value for the SendGrid API key isstored in an Airflow connection.

Store values in Secret Manager

To store values in Secret Manager:

1. Configure Secret Manager for yourenvironment. Make sure to set up permissions and Airflow configurationoptions for the secrets backend.

2. Override the following Airflow configuration options:

   Section	Key	Value
   email	email_conn_id	sendgrid_default
   email	email_backend	airflow.providers.sendgrid.utils.emailer.send_email

3. Create a secret for the SendGrid connection namedairflow-connections-sendgrid_default. Set the secret's value to theconnection URI. For example:

   sendgrid://<username>:<sendgrid_api_key>@smtp.sendgrid.net:587

   For more information about storing connections inSecret Manager, seeManage Airflow connections.

4. Configure the email address for SendGrid. It's not possible to set up theemail address through a secret. Instead,override thefollowing Airflow configuration option:

   Section	Key	Value
   email	from_email	The From email address, such asnoreply@example.com.

   Deprecated: In Airflow versions 2.1.4 to 2.1.3,set theSENDGRID_MAIL_FROMenvironment variable instead ofoverriding the Airflow configuration option.

Store values in Airflow

1. Override the following Airflow configuration options:

   Section	Key	Value
   email	email_conn_id	sendgrid_default
   email	email_backend	airflow.providers.sendgrid.utils.emailer.send_email

2. In Airflow,configure the connectionnamedsendgrid_default. Specify the Sendgrid API key in the connectionURI. For example:

   gcloudcomposerenvironmentsrunENVIRONMENT_NAME\--locationLOCATION\connectionsadd--\--conn-uri"sendgrid://USERNAME:SENDGRID_API_KEY@smtp.sendgrid.net:587"\sendgrid_default

   Replace the following:

   • ENVIRONMENT_NAME: the name of your environment.
   • LOCATION: the region where the environment is located.
   • USERNAME: the SendGrid username.
   • SENDGRID_API_KEY: the SendGrid API key.

3. Configure the email address for SendGrid. It's not possible to set up theemail address through a secret. Instead,override thefollowing Airflow configuration option:

   Section	Key	Value
   email	from_email	The From email address, such asnoreply@example.com.

   Deprecated: In Airflow versions 2.1.4 to 2.1.3,set theSENDGRID_MAIL_FROMenvironment variable instead ofoverriding the Airflow configuration option.

Test your SendGrid configuration

To test SendGrid configuration:

1. Create a test DAG that uses theEmailOperator. For example:

importdatetimeimportairflowfromairflow.operators.emailimportEmailOperatorwithairflow.DAG("composer_sample_sendgrid",start_date=datetime.datetime(2022,1,1),)asdag:task_email=EmailOperator(task_id="send-email",conn_id="sendgrid_default",# You can specify more than one recipient with a list.to="user@example.com",subject="EmailOperator test for SendGrid",html_content="This is a test message sent through SendGrid.",dag=dag,)

1. Upload the DAG to your environment and check that the task succeeds.
2. Sign in to SendGrid in with your SendGrid credentials.
3. In the SendGrid UI, go to Activity page.
4. Search the list for the email. You should see that SendGrid processed and delivered the email.

5. If the email is not processed and delivered:

   • Check your SendGrid configuration.
   • Verify that you enabled the Secret Manager backend. Make sure that you granted extra permissions and set overrides for Airflow configuration options.
   • Check the spam filter in your email client.

Configure third-party SMTP services

To send email through a third-party SMTP service,override theemail_backend Airflow configuration optionand configure other SMTP-related parameters.

To configure a third-party SMTP service,override the following Airflow configuration options:

Configure an SMTP password for a third-party SMTP service

Keeping an SMTP password in plain text in Airflow configuration file is a badsecurity practice. That's why Cloud Composer does not supportthis method. Instead, you can use two other methods for configuring an SMTPpassword.

Using a command to retrieve the SMTP password

It is not possible to retrieve the SMTP password using a command inCloud Composer 1. Providing password through a command will trigger anerror when accessingAdmin>Configuration in theAirflow UI.

Using a secret stored in Secret Manager to retrieve an SMTP password

You canconfigure Secret Manager as yourAirflow secrets backend.

1. Create a new secret:

   echo-n"SMTP_PASSWORD"|gcloudbetasecretscreate\airflow-config-smtp-password\--data-file=-\--replication-policy=automatic

   ReplaceSMTP_PASSWORD with your SMTP password.

2. Configure Airflow to obtain the SMTP password fromSecret Manager. To do so,overridethe following Airflow configuration option:

   Section	Key	Value
   smtp	smtp_password_secret	smtp-password

What's next

• Override Airflow configuration options
• Set environment variables