Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page provides troubleshooting steps and information for common Airflowweb server issues.

The Airflow web server is an Airflow component that provides a user interfacefor managing Airflow DAGs and tasks. This page describes troubleshooting stepsfor various issues with accessing the Airflow web server of your environment orfor web server-related warnings visible in Airflow logs.

Can't access Airflow UI when network access controls are enabled

Symptom: After web server access controls are configured, it's not possibleto access Airflow UI. Usually, the error code displayed in this situation is403.

Information about the issue: Cloud Composer supportsweb server network access controls,which lets you specify IP ranges that are allowed to connect to the web server.

Problems with accessing Airflow UI usually result in error 403. To check if theerror is related to web server network access controls, do the following:

1. In the list of environments, click the name of your environment. TheEnvironment details page opens.
2. Go to theEnvironment configuration tab.
3. Check if theWeb server access control item is set toAll IP addresses have access (default).
4. If a value different thanAll IP addresses have access (default) is configured, then network access control is enabled and the visibility of the Airflow UI is limited to the provided IPv4 and IPv6 address ranges. In this case, the problem might be related to web server network access controls.

In most cases, the cause of the issue is the mismatch between the intended IPthat you specified and the IP that is actually resolved to connect to theAirflow UI. To troubleshoot the problem, do the following:

1. In the list of environments, click the name of your environment.TheEnvironment details page opens.

2. Go to theEnvironment configuration tab.

3. Find theWeb server access control item and clickEdit.

4. In theWeb server network access control dialog, selectAllow access from all IP addresses.

5. Access the Airflow UI multiple times and verify that it works without anyissues:

   • If you don't experience problems, continue to the next step.

   • If you experience a problem at this point, it means that the issue mightbe related to your IAM permissions configuration. Formore information about the IAM permissions forCloud Composer, seeAccess control.

6. In theWeb server network access control dialog, selectAllow access only from specific IP addresses.

7. Add the0.0.0.0/0 IP range, then access the Airflow UI multiple times andverify that it works without issues:

   • If you don't experience problems, then the IP that you're connectingwith is an IPv4 address.

   • If you experience a problem at this point, it means the IP you'reconnecting with is an IPv6 address.

8. Delete the0.0.0.0/0 IP range, and add the::/0 IP range.

   • If you don't experience problems, then the IP that you're connectingwith is an IPv6 address.

   • If you experience a problem at this point, it means the IP you'reconnecting with is an IPv4 address.

9. Now you determined if your resolved address is IPv4 or IPv6.

10. Depending on the address type, narrow down the::/0 or0.0.0.0/0 rangesto more specific ones, to verify the broadest range when the access stopsworking:

    • You can start with the wide subnet mask (such as192.0.2.0/8) thatincludes the address that you assume to be your IP address.

    • To determine your IP address, you can use a third-party service thatprovides your external IP address when you visit its page. You cansearch for such services by the "what is my IP address" search query).

Configuration values aren't displayed on the configuration page

Some Airflow configuration parameters of Airflow are hidden on theconfiguration page to prevent access to potentially sensitive information. Forexample, credentials to access the Airflow database are not displayed.

To display hidden fields,override the following Airflowconfiguration option. We recommend to revert the changes after you obtain therequired values.

DAG crashes the Airflow web server or causes it to return a '502 gateway timeout' error

Web server failures can occur for several different reasons. Checktheairflow-webserver logs inCloud Logging to determine the cause of the502 gateway timeout error.

Heavy load computation

This section applies only to Cloud Composer 1.

Unlike the worker and scheduler nodes, whose machine types can be customized tohave greater CPU and memory capacity, the web server uses a fixed machine type,which can lead to DAG parsing failures if the parse-time computation is tooheavy.

Note that the web server has2 vCPUs and 2 GB of memory.The default value forcore-dagbag_import_timeout is 30 seconds. This timeoutvalue defines the upper limit for how long Airflow spends loading aPython module in the/dags folder.

Incorrect permissions

This section applies only to Cloud Composer 1.

The web server does not run under the same service account as the workers andscheduler. As such, the workers and scheduler might be able to accessuser-managed resources that the web server cannot access.

We recommend that you avoid accessing non-public resources duringDAG parsing. Sometimes, this is unavoidable, and you will need to grantpermissions to the web server's service account. The serviceaccount name is derived from your web server domain. For example, if the domainisexample-tp.appspot.com, the service account isexample-tp@appspot.gserviceaccount.com.

DAG errors

This section applies only to Cloud Composer 1.

The web server runs on App Engine and is separate fromyour environment's GKE cluster. The web server parses the DAGdefinition files, and a502 gateway timeout can occur if there are errorsin the DAG. Airflow works normally without a functional web server if theproblematic DAG is not breaking any processes running in GKE.In this case, you can usegcloud composer environments run to retrievedetails from your environment and as a workaround if the web server becomesunavailable.

In other cases, you can run DAG parsing in GKE and look forDAGs that throw fatal Python exceptions or that time out (default 30 seconds).To troubleshoot, connect to a remote shell in an Airflow worker container andtest for syntax errors. For more information,seeTesting DAGs.

What's next

• Access Airflow UI
• Access control