This page describes how to troubleshoot errors you might encounter while using Cloud Run.Personalized Service Health publishesall Cloud Run incidents that stem from theunderlying Google Cloud infrastructure to identify Google Cloud service disruptionsimpacting your projects. You should also considersetting up alerts on Personalized Service Health events. For informationabout incidents affecting all Google Cloud services, see theGoogle Cloud Service Health dashboard.

Check for existing issues or open new issues in thepublic issue trackers.

For other errors messages not listed In this page, seeKnown issues in Cloud Run. If you continue to encounter errors even after followingthe steps in this guide,contact support.

See the following sections for guidance on how to resolve issues in Cloud Run:

• Deployment errors
• Serving errors
• Connectivity and security errors

Deployment errors

This section describes the common deployment errors in Cloud Run andmethods to troubleshoot them.

Container failed to start

The following error occurs when you try to deploy:

Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable.

To resolve this issue, follow these steps:

1. Verify that you canrun your container image locally. If yourcontainer image cannot run locally, you need to diagnose and fix the issuelocally first.

2. Check if your container islistening for requests on the on the correct port.Your container must listen for incoming requests on the port that is definedby Cloud Run and provided in thePORT environment variable. Forinstructions on how to specify the port,seeConfiguring containers for services

3. Check if your container is listening on allnetwork interfaces, commonly denoted as0.0.0.0. Notably, your containershouldnot listen on127.0.0.1.

4. Verify that your container image is compiledfor 64-bit Linux as required by thecontainer runtime contract.

   Note: If you build your container image on a ARM based machine, then itmight not work as expected when used with Cloud Run. To solvethis issue,build your image using Cloud Build.

5. Use Cloud Logging tolook for application errors instdout orstderr logs.You can alsolook for crashes captured in Error Reporting.

   You might need to update your code or your revision settings to fix errorsor crashes. You can alsotroubleshoot your service locally.

Container import error

The following error occurs when you try to deploy:

The service has encountered an error during container import. Please try again later. Resource readiness deadline exceeded.

To resolve this issue, follow these steps:

1. Ensure container's file system does not containnon-utf8 characters.

2. Some Windows based Docker images make use offoreign layers.Cloud Run's control plane doesn't support foreign-layers. Toresolve this issue, try setting the--allow-nondistributable-artifacts flagin the Docker daemon.

The feature is not supported

The following error occurs when you call the Cloud Run Admin API:

The feature is not supported in the declared launch stage

This error occurs when you call the Cloud Run Admin API directly and use a betafeature without specifying a launch stage annotation or field.

To resolve this issue, add the launch stage field to the requests.

Refer to the following examples for adding launch stage references when usingthe v1 or the v2 REST API:

• Launch stage annotation to a client request usingJSON and the v1 REST API:

  "annotations":{"run.googleapis.com/launch-stage":"BETA"}

• LaunchStage reference to aclient request usingJSON and the v2 REST API:

  "launchStage":"BETA"

• Launch stage annotation to a service request usingYAML and the v1 REST API:

  kind:Servicemetadata:annotations:run.googleapis.com/launch-stage:BETA

Userroot is not found

The following error occurs when the customer managed encryption keys arespecified using a--key parameter:

ERROR: "User \"root\""not found in /etc/passwd

To resolve this issue, specifyUSER 0 instead ofUSER root in theDockerfile.

Default Compute Engine service account is deleted

The following error occurs during deployment:

ERROR: (gcloud.run.deploy) UserEMAIL_ADDRESS does not have permission to access namespaceNAMESPACE_NAME (or it may not exist): Permission 'iam.serviceaccounts.actAs' denied on service accountPROJECT_NUMBER-compute@developer.gserviceaccount.com (or it may not exist).

This issue occurs in either of the following scenarios:

• ThedefaultCompute Engine service accountdoesn't exist in the project, and a service account isn't specified with the--service-account flag at the time of deployment.

• The developer or principal deploying the service is doesn't have therequired permissionsfor the defaultCompute Engine service account to deploy.

To resolve this issue:

1. Specify a service account using the--service-account flag:

   gcloud run services updateSERVICE_NAME --service-accountSERVICE_ACCOUNT

2. Verify that the service account you specify has thepermissions required to deploy.

To verify if the default Compute Engine service agent exists in your Google Cloud project, follow these steps:

1. In the Google Cloud console, go to the Identity and Access ManagementPermissions page:

   Go to Permissions

2. Select theInclude Google-provided role grants checkbox.

3. In thePrincipals list, locate the ID of the Compute Engine serviceagent, which follows the format,PROJECT_NUMBER-compute@developer.gserviceaccount.com.

Issues with Cloud Build service account

The following error occurs during source deployments when the Cloud Buildservice account doesn't have the required permissions or is disabled:

ERROR: (gcloud.run.deploy) NOT_FOUND: Build failed. The service has encountered an internal error. Please try again later. This command is authenticated asEMAIL_ADDRESS which is the active account specified by the [core/account] property.

Cloud Build changed the default behavior for how Cloud Build usesservice accounts in new projects. This is detailed inCloud Build default service account change. As a result of this change, new projectsdeploying to Cloud Run from source code for the first time might beusing a default Cloud Build service account with insufficient permissions fordeploying from source.

To resolve this issue, follow these steps:

• Review the Cloud Build guidance on changes to the default service account andopt outof these changes.

• Grant theCloud Run Builder(roles/run.builder) role to the build service account.

Cloud Run Service Agent missing permission to read the image

The following error occurs when you try to deploy from a projectusing an image that is stored in the Artifact Registry, using thegcr.io domainin a different project:

Google Cloud Run Service Agent must have permission to read the image, gcr.io/PROJECT-ID/IMAGE-NAME. Ensure that the provided container image URL is correct and that above account has permission to access the image. If you just enabled the Cloud Run API, the permissions might take a few minutes to propagate. Note thatPROJECT-ID/IMAGE-NAME is not in projectPROJECT-ID-2. Permission must be granted to the Google Cloud Run Service Agent from this project.

You might also see the following error when you try to deploy from a projectusing an image that is stored in the Artifact Registry in a different project:

ERROR: (gcloud.run.deploy) PERMISSION_DENIED: User must have permission to readthe image,REGION.pkg.dev/PROJECT_ID/ARTIFACT_REGISTRY_REPO/IMAGE:latest. Ensure that the provided container image URL is correctand that the above account has permission to access the image. If you just enabledthe Cloud Run API, the permissions might take a few minutes to propagate. Notethat the image is from projectPROJECT_ID, which is not the same asthis projectPROJECT_ID.

To resolve this issue, follow these troubleshooting recommendations:

• Follow the instructions fordeploying container images from other Google Cloud projectsto ensure that your principals have the necessary permissions.

• This issue might also occur if the project is in a VPC Service Controlsperimeter with a restriction on the Cloud Storage API that prohibitsrequests from the Cloud Run service agent. To fix this:

  1. Open Logs Explorer in the Google Cloud console. (Do not use theLogs page inside the Cloud Run page):

     Go to Logs Explorer

  2. Enter the following text in the query field:

     protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"severity=ERRORprotoPayload.status.details.violations.type="VPC_SERVICE_CONTROLS"protoPayload.authenticationInfo.principalEmail="service-PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com"

  3. If you see any log entries after you use this query, examine the logentries to determine whether you need to update your VPC Service Controls policies.They may indicate that you need to addservice-PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.comto a pre-existing access policy.

Missing permissions for source deployments

The following errors might occur when deploying from the source:

ERROR: (gcloud.run.deploy)EMAIL_ADDRESS does not have permissionto access namespaces instancePROJECT_ID (or it may not exist): GoogleCloud Run Service Agent does not have permission to get access tokens forthe service accountSERVICE_ACCOUNT. Please giveSERVICE_ACCOUNTpermissioniam.serviceAccounts.getAccessToken on the service account.Alternatively, if the service account is unspecified or in the same project youare deploying in, ensure that the Service Agent is assigned the GoogleCloud Run Service Agent roleroles/run.serviceAgent. Thiscommand is authenticated asEMAIL_ADDRESS, which is the active accountspecified by the [core/account] property.

Every Cloud Run service is associated with a service account thatserves as itsidentity whenthe service accesses other resources. This service accountmight be the default service account (PROJECT_NUMBER-compute@developer.gserviceaccount.com) or a user-managed service account.

In environments where multiple services are accessing different resources, youmight use per-service identities with different user-managed service accounts instead ofthe default service account.

To resolve this issue, grant the deployer account theService Account User role (roles/iam.serviceAccountUser)on the service account that is used as the service identity.This predefined role contains theiam.serviceAccounts.actAs permission, whichis required to attach a service account on the service or revision. A user whocreates a user-managed service account is automatically granted theiam.serviceAccounts.actAs permission, however, other deployers must have thispermission granted by the user who creates the user-managed service account.

For more information regarding access requirements for any new service accountsyou create, seeGet recommendations to create dedicated service accounts.

User has insufficient permissions to complete source deployments

The following error occurs when the deployer account is missing the requiredpermissions on your project:

ERROR: (gcloud.run.deploy) 403 Could not upload fileEMAIL_ADDRESS doesnot havestorage.objects.create access to the Google Cloud Storage object. Permissionstorage.objects.create denied on resource (or it may not exist). Thiscommand is authenticated asEMAIL_ADDRESS which is the active account.

To resolve this error, ask your administrator to grant you the following Identity and Access Management roles:

• Cloud Run Source Developer (roles/run.sourceDeveloper) on your project.
• Service Usage Consumer (roles/serviceusage.serviceUsageConsumer) on your project.
• Service Account User (roles/iam.serviceAccountUser) on the Cloud Run service identity. For moreinformation, seeDeployment permissions.

Error when deploying a Cloud Run service from other Google Cloud projects

The following error occurs when you deploy a Cloud Run service froma source project to a target project:

Failed to create service.Operation failed due to missing permissions.Google Cloud Run Service Agent does not have permission to get accesstokens for the service accountSERVICE_ACCOUNT. Please giveSERVICE_ACCOUNT permissioniam.serviceAccounts.getAccessTokenon the service account. Alternatively, if the service account is unspecified orin the same project you are deploying in, ensure that the Service Agent isassigned the Google Cloud Run Service Agent roleroles/run.serviceAgent.

To resolve this issue, follow these steps:

1. Grant theService Account User (roles/iam.serviceAccountUser) role on the service account that you use as the service identity in your target project.

2. Grant theService Account Token Creator (roles/iam.serviceAccountTokenCreator) role to the Cloud Run service account in the target project. Add the Cloud Runservice agent email (service-PROJECT_NUMBER@SERVICE_DOMAIN.iam.gserviceaccount.com) as the principal from the source project.

3. Turn off theiam.disableCrossProjectServiceAccountUsage organization policy.

For detailed instructions, seeConfigure service identity for services.

Errors when deploying your Python source code to Cloud Run

When you deploy a Cloud Run service from source code with the Pythonruntime, one of the following errors occurs:

• RevisionREVISION_NAME is not ready and cannot serve traffic.The user provided container failed to start and listen on port defined by PORT=8080environment variable within the allocated timeout. This can happen if the port ismisconfigured or if the timeout is too short. The healthcheck timeout can be extended.

• Deployment succeeds with HTTP 500 error codes in the logs.

The Python buildpack sets the default entrypoint forCloud Run source deployments. For Python version 3.13and later, the Python buildpack sets the entrypoint based on the web service configuration in yourrequirements.txt file. If you don't specify aweb server or framework in therequirements.txt file, or use Python version 3.12 and earlier, the Python buildpack sets the default entrypoint togunicorn -b :8080 main:app. For more information, seeBuilding a Python application.

You can resolve this issue in several ways. For example, you can specify one of the following web servers in yourrequirements.txt file:

• gunicorn:

    # https://pypi.org/project/gunicorn/  gunicorn==21.2.0

• fastapi anduvicorn:

  #https://pypi.org/project/fastapifastapi[standard]==0.116.1#https://pypi.org/project/uvicornuvicorn==0.35.0

Alternatively, you can follow either of these steps to resolve deployment errors:

• Specify the entrypoint by running the following source deploy command:

  gcloud run deploySERVICE --source .  --set-build-env-vars GOOGLE_ENTRYPOINT="ENTRYPOINT"

  Replace the following:

  • SERVICE: the name of the service you want todeploy to.
  • ENTRYPOINT: the default entrypoint you want to use for your sourcecode.

• Use aProcfile to set anentrypoint.

Serving errors

This section lists issues that you might encounter with serving and providessuggestions for how to fix each of them.

HTTP 404: Not Found

The following issue occurs during serving:

`HTTP 404`:Not found

You might encounterHTTP 404 errors in the following scenarios:

1. An incorrect request URL or application code returns404 errors. To resolvethis issue, follow these steps:

   1. Check if the URL you requested is correct. You can verify the URL either inthe service details page in the Google Cloud console or by running the following command:

      gcloudrunservicesdescribeSERVICE_NAME|grepURL

   2. Inspect where your app returns404 error codes. If yourapp returns404, you can find it in Cloud Logging. Also, verify thatthe app does not return a404 error code when yourun it locally.

   3. Make sure that your app doesn't start listening on its configured port before itis ready to receive requests.

2. The request doesn't reach the container, leading to a404 error in the followingscenarios:

   • A request doesn't satisfy the specified network restriction, especiallywhen Cloud Run service'singress settingsare set toInternal orInternal and Cloud Load Balancing.

   • The Cloud Run service'sdefaultrun.app URL is disabled, and a client attempts to reach theservice at thatrun.app URL.

   In both of these scenarios, you can't find the404 error inCloud Logging, even when you apply the following filter:

   resource.type="cloud_run_revision"log_name="projects/PROJECT_ID/logs/run.googleapis.com%2Frequests"httpRequest.status=404

3. With the same ingress settings, the request might be blocked byVPC Service Controls based on thecaller's context, including project and IP address. To check for a VPC Service Controlspolicy violation:

   1. Open Logs Explorer in the Google Cloud console:

      Go to Logs Explorer

   2. Enter the following text in the query field:

      resource.type="audited_resource"log_name="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fpolicy"resource.labels.method="run.googleapis.com/HttpIngress"

   3. If you see any log entries after you use this query, examine the logentries to determine whether or not you need to update your VPC Service Controlspolicies.

4. Access your service endpoint with a load balancer using the Python runtime.To resolve this issue,verify the URL mask for your load balancer, and ensure that theURL path you specify for the load balancer matches the path in your Python source code.

No available container instances

The following error occurs during serving:

HTTP 429The request was aborted because there was no available instance.The Cloud Run service might have reached its maximum container instancelimit or the service was otherwise not able to scale to incoming requests.This might be caused by a sudden increase in traffic, a long container startup time or a long request processing time.

To resolve this issue, check theContainer instance count metric for yourservice and consider increasing this limit if your usage is nearing the maximum.For more information, seeSet maximum instances for services, and if you need more instances,request a quota increase.

Cloud Run couldn't manage the rate of traffic

The following error occurs either during serving or when the servicehasn't reached itsmaximum container instance limit:

HTTP 500The request was aborted because there was no available instance

To resolve this issue, follow these steps:

1. Address the following possible root causes:

   • A huge sudden increase in traffic.
   • A longcold start time.
   • A long request processing time, or a sudden increase in requestprocessing time.
   • The service reaching its maximum container instance limit (HTTP 429).
   • Transient factors attributed to the Cloud Run service.

2. Implement exponential backoff and retries for requests that the client mustnot drop. A short and sudden increase in traffic or requestprocessing time might only be visible in Cloud Monitoring if you zoomin to a 10-second resolution.

3. When the root cause of the issue is a period of heightened transient errorsattributable solely to Cloud Run, contactsupport.

Cloud Run instance fails to start

The following error occurs during serving:

HTTP 500The request failed because the instance could not start successfully.

This error occurs when the container fails to start and become healthy withinyour configured time. While this can have multiple causes, a common reason isthe container's inability to access a required secret fromSecret Manager.

To resolve this issue, consider the following:

1. Verify IAM permissions: Ensure that your Cloud Runservice account has theSecret Manager Secret Accessor (roles/secretmanager.secretAccessor) role on the secretyou are trying to access.

2. Check the secret path: Verify that youspecify the correct secret name andversion in yourapplication's configuration.

3. Inspect network configuration: If you are using a VPC Service Controls perimeteror firewall rules, ensure that youallow egresstraffic tosecretmanager.googleapis.com.

4. Output effective logs: Addlogging to yourapplication's startup process, especially around fetching secrets, todiagnose specific failures.

Operation is not implemented

The following error occurs when you specify an incorrectREGION whileinvoking your Cloud Run job, such as when you deploy a job in theregionasia-southeast1, and invoke your job usingsoutheast1-asia orasia-southeast:

HTTP 501Operation is not implemented, or supported, or enabled.

For the list of supported regions, seeCloud Run locations.

Default credentials not found

The following error occurs when your application isn't authenticated properlydue to missing files, invalid credential paths, or incorrect environment variablesassignments:

HTTP 503: System.InvalidOperationException System.InvalidOperationException your Defaultcredentials were not found.

To resolve this issue:

1. Install and initialize the gcloud CLI.

2. Set up Application Default Credentials (ADC) with the credentials that areassociated with the user. Configure ADC using:

     gcloud auth application-default login

   A sign-in screen appears. After you sign in, your credentials are stored inthelocal credential file,used by ADC.

3. Use theGOOGLE_APPLICATION_CREDENTIALS environment variable to provide thelocation of a credential JSON file within your Google Cloud project.

   For more information, seeSet up Application Default Credentials.

Container instances exceed memory limits

The followingHTTP 500 orHTTP 503 error occurs during serving in Cloud Logging:

While handling this request, the container instance was found to be using too much memory and was terminated. This is likely to cause a new container instance to be used for the next request to this revision. If you see this message frequently, you may have a memory leak in your code or may need more memory. Consider creating a new revision with more memory.

To resolve this issue:

1. Determine if your container instances are exceeding the available memory.Look for related errors in thevarlog/system logs.
2. If the instances are exceeding the available memory, considerincreasing the memory limit.

In Cloud Run, files written to the local file system counttowards the available memory. This also includes any log files that are writtento locations other than/var/log/* and/dev/log.

Unable to process some requests due to high concurrency setting

The following error occurs when your container instances are using a highCPU load to process requests, and as a result, are unable to process all ofthe requests:

HTTP 503The Cloud Run service probably has reached its maximum container instance limit. Consider increasing this limit.

To resolve this issue, follow these steps:

1. Increase the maximum number of container instancesfor your service.

2. Lower the service'sconcurrency. For moreinformation, seeSet maximum concurrent requests per instance.

Cloud Logging errors related to container restrictions

When running under root namespaces that violate constraints listed in thecontainer runtime contract,you might see Cloud Logging logs that contain errors from the Linux kernelsuch as "Operation not permitted" or "Permission denied." These errors appearwhen the system calls attempting to perform these actions are rejected. Forguidance on making your containers more compatible with Cloud Runexecution environments, seeAdapt your containers for Cloud Run.

Attempts to mount file systems might have logs such as the following:

• mount.nfs: Operation not permitted
• mount.cifs: Operation not permitted

If your application tries to modify files under/proc or/sys with toolslikesysctl, your logs might contain errors such as the following:

• sysctl: permission denied on key "vm.max_map_count"

Attempts to runsudo will fail with logs that declare the following:

• sudo: The "no new privileges" flag is set

Other system calls, likemlockall, might fail with errors such as thefollowing:

• msg="mlockall syscall failed" error="cannot allocate memory"

Errors are unique to the code in question and to how well you implement loggingand error handling.

Cloud Logging errors related to pending queue request aborts

One of the following errors occur when Cloud Run fails to scaleup fast enough to manage traffic:

• The request was aborted because there was no available instance:severity=WARNING ( Response code: 429 ) Cloud Run cannotscale due to themax-instances limit you setduring configuration.

• severity=ERROR ( Response code: 500 ) Cloud Run intrinsicallycannot manage the rate of traffic.

To resolve this issue, follow these steps:

1. Address the root causes that might cause scaling failures, such as:

   • A huge sudden increase in traffic.
   • Longcold start time.
   • Long request processing time.
   • High source code error rate.
   • Reaching themaximum instance limit and preventing the system from scaling.
   • Transient factors attributed to the Cloud Run service.

   For more information on resolving scaling issues, and optimizing performance, seeGeneral development tips.

2. For HTTP trigger-based services or functions, have the client implement exponentialbackoff and retries for requests that must not be dropped. If you are triggeringservices from Workflows, you can use thetry/retry syntax to achieve this.

3. For background or event-driven services or functions, Cloud Run supportsat-least-once delivery.Even without explicitly enabling retry, Cloud Run automaticallyre-delivers the event and retries the execution. SeeRetrying event-driven functionsfor more information.

4. For issues pertaining to cold starts, configureminimum instances to reduce the amount of cold starts with a higher billing implication.

5. When the root cause of the issue is a period of heightened transient errorsattributed solely to Cloud Run or if you need assistance withyour issue,contact support.

Identity token signature redacted by Google

The following error occurs during development and testing phases:

SIGNATURE_REMOVED_BY_GOOGLE

This error might occur in the following scenarios:

• The user logs in by using Google Cloud CLI or Cloud Shell.

• The user generates an ID token by usinggcloud commands.

• The user tries to use the ID token to invoke a non-public Cloud Runservice.

This is the expected default behavior. Google removes the token signature dueto security concerns to prevent any non-public Cloud Run service fromreplaying ID tokens that are generated in this manner.

To resolve this issue, invoke your private service with a new ID token. For moreinformation, seeTest your private service.

OpenBLAS warning in logs

If you useOpenBLAS-basedlibraries such as NumPy with thefirst generation execution environment,you might see the following warning in your logs:

OpenBLAS WARNING - could not determine the L2 cache size on this system,assuming 256k`

The OpenBLAS warning occurs when the container sandbox used by the first generationexecution environment doesn't expose low level hardware features. This warningdoesn't impact your service. To avoid OpenBLAS warnings log entries,switch to the second generation execution environment.

Spark fails when obtaining IP address of machine to bind to

When Spark fails while obtaining the IP address of the binding machine, one ofthe following errors occurs:

assertion failed: Expected hostname (not IP) but got <IPv6 ADDRESS>

assertion failed: Expected hostname or IPv6 IP enclosed in [] but got <IPv6 ADDRESS>

To resolve this issue, set theSPARK_LOCAL_IP environment variable to127.0.0.1 inyour Dockerfile, for example,ENV SPARK_LOCAL_IP="127.0.0.1". If you don'tset theSPARK_LOCAL_IP environment variable, it defaults to its IPv6 counterpartinstead of the localhost. Additionally, Spark doesn't recognize the environmentvariable set asRUN export SPARK_LOCAL_IP="127.0.0.1".

Cannot access files using NFS

See the Cloud Storage FUSEtroubleshooting guide.

High latency when CPU utilization is low

Your service might experience high request latencies or fail to scale up underload, even though Cloud Monitoring shows average CPU utilization is wellbelow the typical 60% scaling target.

Potential cause:

This might occur if your application is single-threaded for CPU-boundtasks but is deployed on an instance with multiple vCPUs. Your applicationmight max out one vCPU core while other cores remain mostly idle. The Cloud Runautoscaler uses the average CPU utilization across all vCPUs; this average mightbe low in these cases, which prevents CPU-based scaling.

Solutions:

• For single-threaded applications:
  • Prefer configuring your service with 1 vCPU if its memory requirements can be met (seeMemory limits and CPU minimums). This helps CPU utilization metrics accurately reflect load.
  • If multiple vCPUs are necessary due to high memory needs that exceed 1-vCPU limits, CPU-based autoscaling will likely be less effective. In this scenario, lower the max concurrency setting to encourage scaling sooner based on request throughput. SeeConcurrency Configuration.
• For multi-vCPU configurations: Ensure your application is architected to effectively utilize all allocated vCPUs (for example, using multiple worker processes or threads).

Connectivity and security errors

This section describes the common connectivity and security errors inCloud Run and methods to troubleshoot them.

Client is not authenticated properly

The following error occurs during serving:

HTTP 401: The request was not authorized to invoke this service

To resolve this issue:

1. If aservice account invokes your Cloud Run service, setthe audience claim (aud) of the Google-signed ID token to the following:

   • If you set theaud to the URL of the receiving service using the formathttps://SERVICE.run.app orhttps://REGION-PROJECT_ID.cloudfunctions.net/FUNCTION, your service mustrequire authentication. Invoke your Cloud Run service using the Cloud Run URL or through aload balancer URL. For examples on sending authenticated requests, seeInvoke with an HTTPS request.

   • If you set theaud to the Client ID of an OAuth 2.0 Client ID with typeWeb application, using the formatnnn-xyz.apps.googleusercontent.com,you can invoke your Cloud Run service through anHTTPS load balancer secured by IAP. We recommendthis approach for an Application Load Balancer backed by multipleCloud Run services in different regions.

   • If you set theaud to a configuredcustom audience, use the exact values provided. For example,if the custom audience ishttps://service.example.com,the audience claim value must also behttps://service.example.com.

2. Make sure that your requests include anAuthorization: Bearer ID_TOKENheader, or anX-Serverless-Authorization: Bearer ID_TOKEN header for custom authorization, and that the token is an ID token, not an access or refreshtoken. A401 error might occur in the following scenarios due to incorrectauthorization format:

   • Theauthorization tokenuses an invalid format.

   • The authorization header isn't aJSON Web Token (JWT) with a valid signature.

   • The authorization header contains multiple JWTs.

   • Multiple authorization headers are present in the request.

   To check claims on a JWT, use thejwt.io tool.

3. If you get invalid tokens when using themetadata server to fetch ID and access tokens to authenticate requests with the Cloud Run service or job identity with an HTTP proxy to route egress traffic, add the following hosts to theHTTP proxy exceptions:

   • 169.254.* or169.254.0.0/16

   • *.google.internal

4. A401 error commonly occurs whenCloud Client Librariesuse the metadata server to fetchApplication Default Credentials to authenticate REST or gRPC invocations.If you don't define the HTTP proxy exceptions, the following behavior results:

   • If different Google Cloud workloads host a Cloud Run service or job and the HTTP proxy, even if the Cloud Client Librariesfetch the credentials, the service account that's assigned to the HTTP proxy workload generates the tokens. The tokens might nothave the required permissions to perform the intendedGoogle Cloud APIoperations. This is because the service account fetches the tokens from the view of theHTTP proxy workload's metadata server, instead of the Cloud Run service or job.

   • If the HTTP proxy isn't hosted in Google Cloud, and you route metadata serverrequests using the proxy, then the token requests fail andtheGoogle Cloud APIs operations don't authenticate.

5. Redeploy your service toallow public access if this is supported by your organization. This is useful for testing.

Client is not authorized to invoke the service

One of the following errors occur while invoking your service:

HTTP 403: The request was not authenticated. Either allow public access or set the proper Authorization header

HTTP 403: Forbidden: Your client does not have permission to get URL from this server.

A403 error might occur when the IAMmember used to generate the authorization token is missing therun.routes.invokepermission. Grant this permission to the user generating the token.

Additionally, if there is an entry for the error with the formatresource.type = "cloud_run_revision" in Cloud Logging, follow these steps to resolve the error:

1. To make your service invocable by anyone,update the IAM settings, and make the service public.

2. To make your service invocable only by certain identities,invoke your service with the proper authorization token:

   • If adeveloper oran enduser invokes your service, grant therun.routes.invoke permission. You can provide thispermission through the Cloud Run Admin (roles/run.admin) and Cloud Run Invoker (roles/run.invoker) roles.

   • If aservice account invokes your service,ensure that the service account is a member of the Cloud Runservice, and grant the Cloud Run Invoker (roles/run.invoker)role.

   • Calls missing an auth token might cause the403 error. If calls witha valid auth token still lead to the403 error, grant the IAMmember that generates the token therun.routes.invoke permission.

When you encounter a403 error and don't find the log entryresource.type = "cloud_run_revision", it might be due toVPC Service Controls blocking a Cloud Run service that hasingress settings configured toAll. For more information ontroubleshooting VPC Service Controls denials, see404 errors.

Error when accessing the service from a web browser

The following issue occurs when you access a Cloud Run service from a webbrowser:

403 ForbiddenYour client does not have permission to get URL from this server.

When you invoke a Cloud Run service from a web browser, the browsersends aGET request to the service. However, therequest doesn't contain the authorization token of the calling user. To resolvethis issue, follow these steps:

1. UseIdentity-Aware Proxy (IAP)with Cloud Run. IAP lets you establish acentral authorization layer for applications accessed over HTTPS. WithIAP, you can use an application-level access control modelinstead of network-level firewalls. For more details on configuringCloud Run with IAP, seeEnabling Identity-Aware Proxy for Cloud Run.

2. As a temporary workaround, access your service through a web browser using theCloud Run proxy in Google Cloud CLI. To proxy a servicelocally, run the following command:

   gcloud run services proxySERVICE --projectPROJECT-ID

   Cloud Run proxies the private servicetohttp://localhost:8080 (or to the port you specify with--port), providingthe token of the active account or another token you specify.This is the recommended way to test privately a website or API in your browser.For more information, seeTesting private services.

3. Allow public access to yourservice. This is helpful for testing, or when your service is apublic API or website.

Connection reset by peer

One of the following errors occurs when a peer across the network unexpectedly closesthe TCP connection established by the application:

Connection reset by peer

asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation

grpc.StatusRuntimeException: UNAVAILABLE: io exception

psycopg.OperationalError: the connection is closed

ECONNRESET

To resolve this issue, follow these steps:

• If you are trying to perform background work with CPU throttling, usetheinstance-based billingsetting.

• Ensure that you are within theoutbound requests timeouts.If your application maintains any connection in an idle state beyond thisthreshold, the gateway needs to reap the connection.

• By default, the TCP socket optionkeepalive is disabled forCloud Run. There is no direct way to configure thekeepalive option at the service level.To enable thekeepalive option for each socket connection,provide the required socket options when opening a new TCP socket connection,depending on the client library you are using for this connection inyour application.

• Occasionally,outbound connections are resetdue to infrastructure updates. If your application reuses long-livedconnections, we recommend that you configure your application tore-establish connections to avoid the reuse of a dead connection.

• If you're using an HTTP proxy to route your Cloud Run services orjobs egress traffic, and the proxy enforces maximum connection duration,the proxy might silently drop long-running TCP connections such as the onesestablished using connection pooling. This causes HTTP clients to fail whenreusing an already closed connection.If you intend to route egress traffic through an HTTP proxy, you must implementconnection validation, retries, and exponential backoff. For connection pools, configure maximum values for connection age, idle connections, and connection idle timeout.

Connection timeouts

The following errors occur when an application attempts to create a new TCPconnection with a remote host, and the connection takes too long toestablish:

java.io.IOException: Connection timed out

ConnectionError: HTTPSConnectionPool

dial tcpREMOTE_HOST:REMOTE_PORT: i/o timeout / context error

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

To resolve connection timeout issues, follow these steps:

1. If you're routing all egress traffic through a VPC network,either usingVPC connectorsorDirect VPC egress, follow these steps:

   • Define all the necessary firewall rules to allow ingresstraffic for theVPC connectors.

   • VPC firewall rules mustallow ingresstraffic from the VPC connectors or the Direct VPC egresssubnet to reach the destination host or subnet.

   • Be sure that you have all the required routes to allow correct traffic routing to thedestination hosts and back. This is important when routing egresstraffic throughVPC Network Peering orhybrid cloud connectivity, as packets traversemultiple networks before reaching the remote host.

2. If you're using an HTTP proxy to route all egress traffic from yourCloud Run services or jobs, the remote hosts must bereachable using the proxy.

   Traffic routed through an HTTP proxy might be delayed depending on theproxy's resource utilization. If you are routing HTTP egress traffic usinga proxy, implement retries, exponential backoff, or circuit breakers.

Configure HTTP proxy exceptions

When using an HTTP proxy to route your Cloud Run services or jobsegress traffic, add exceptions forCloud APIs,and other non-proxied hosts and subnets, to prevent latency,connection timeouts, connection resets and authentication errors.

Non-proxied hosts and subnets must include, at minimum, the following:

• 127.0.0.1
• 169.254.* or169.254.0.0/16
• localhost
• *.google.internal
• *.googleapis.com

Optionally, non-proxied hosts might include:

• *.appspot.com
• *.run.app
• *.cloudfunctions.net
• *.gateway.dev
• *.googleusercontent.com
• *.pkg.dev
• *.gcr.io

To set HTTP proxy exceptions for egress networking, configure the following:

• Environment variables:NO_PROXY orno_proxy.

• Java Virtual Machine flaghttp.nonProxyHosts:

  • The system propertyhttps.nonProxyHosts isn't defined. Thissystem property applies to both HTTP and HTTPS.

  • The system propertyhttp.nonProxyHosts doesn't supportCIDR notation. You must use pattern matching expressions.

Malformed response or container instance connection issue

The following error occurs when there is a container instance connection issue:

HTTP 503The request failed because either the HTTP response was malformed or connection to the instance had an error.

To resolve this issue, follow these steps:

1. CheckCloud Loggingfor the following errors:

   • Out of memory errors. If the logs contain error messages regarding containerinstances exceeding memory limits, see the recommendations in theContainer instances are exceeding memory limitssection.

   • Liveness probe failureswith the following error in the logs:

     LIVENESS HTTP probe failed 1 time consecutively for containerCONTAINER_NAME on port 8080. The instance has been shut down.

     When your instance fails to respond successfully to the probe within thetimeout period, follow these steps:

     • Enable instrument logging andtracing to determine the cause of the increased latency.

     • Increase the timeout periodfor the liveness probe.

2. If requests are terminating with error code503before reaching therequest timeout set in Cloud Run,update the request timeout setting for your language framework:

   • Node.js developers must update theserver.timeout property throughserver.setTimeout (useserver.setTimeout(0) to achieve an unlimited timeout) dependingon the version you are using.

   • Python developers need to updateGunicorn's default timeout ([CRITICAL] WORKER TIMEOUT).

3. In some scenarios, a503 error code might occur due to a downstream networkbottleneck, such as during load testing. Forexample, if your service routes traffic through aServerless VPC Access connector, ensure that the connectordoesn't exceed its throughput threshold by following these steps:

   1. Open Serverless VPC Access in the Google Cloud console:

      Go to Serverless VPC Access

   2. Check for anyred bars in the throughputchart histogram. If there is ared barconsider increasing the max instances or instance type your connectoruses. Alternatively, compress traffic sent through aServerless VPC Access connector.

4. If a container instance receives more than800 requests per second, the available TCP sockets might be exhausted. To resolve this,turn on HTTP/2 for your service, and make the required changes to your service tosupport HTTP/2.

Gateway timeout error

The following error occurs when yourservice doesn't return a response within a specified time, and the request ends:

HTTP 504The request has been terminated because it has reached the maximum request timeout.

For more information about this error, see theContainer runtime contract.

To troubleshoot this issue, follow these steps:

• If your service is processing long requests,increase the request timeout.

• Instrument logging andtracing to understand where yourapp is spending time before exceeding your configured request timeout.

• Outbound connections are resetoccasionally, due to infrastructure updates. If your application reuses long-livedconnections, we recommend that you configure your application tore-establish connections to avoid the reuse of a dead connection.

  Depending on your app's logic or error handling, a504 error might be a signalthat your application is trying to reuse a dead connection and therequest blocks until your configured request timeout. Use aliveness probe to terminate an instance thatreturns persistent errors.

• Out of memory errors that happen inside the app's code,for example,java.lang.OutOfMemoryError, don't necessarily terminate acontainer instance.If memory usage doesn't exceed thecontainer memory limit, then Cloud Run won't terminate the instance. Depending on how the apphandles app-level out of memory errors, requests might not go through untilthey exceed your configured request timeout.

  To terminate the container instance, follow these steps:

  • Configure your app-level memory limit to be greater than yourcontainer memory limit.

  • Use a liveness probe to help terminate an instance that returns persistenterrors.

Custom domain stuck while provisioning certificate

One of the following errors occurs when you map a custom domain:

The domain is available over HTTP.  Waiting for certificate provisioning. You must configure your DNS records for certificate issuance to begin and to accept HTTP traffic.

Waiting for certificate provisioning. You must configure your DNS records for certificate issuance to begin.

To resolve this issue:

1. Wait at least 24 hours. Provisioning the SSL certificate usually takes about15 minutes, but it can take up to 24 hours.

2. Verify that you've properly updated your DNS records at your domainregistrar using theGoogle Admin Toolbox dig tool. The DNS records in your domain registrar must match what theGoogle Cloud consoleprompts you to add.

3. Verify theroot of the domain under your account usingone of the following methods:

   • Follow the instructions foradding verified domain owners, and check that your account is listed as aVerified Owner.

   • Use theSearch Console.

4. Verify that the certificate for the domain is not expired. To find theexpiry bounds, use the following command:

   echo | openssl s_client -servername 'ROOT_DOMAIN' -connect 'ROOT_DOMAIN:443' 2>/dev/null | openssl x509 -startdate -enddate -noout

Client disconnect does not propagate to Cloud Run

When you use HTTP/1.1 on Cloud Run, client disconnect events are notpropagated to the Cloud Run container.

To resolve this issue, use Websockets or HTTP/2.0, which propagate client disconnects.