This document shows you how to resolve issues with the Compute Engine metadataserver.

Compute Engine VMs store metadata on a metadata server. VMsautomatically have access to the metadata server API without any additionalauthorization. However, sometimes VMs might lose access to the metadata serverdue to one of the following causes:

• Failure to resolve the metadata server domain name
• Connection to the metadata server is blocked by one of the following:
  • OS level firewall configuration
  • Proxy setup
  • Custom routing

When VMs can't access the metadata server, some processes might fail.

For information about troubleshooting thegke-metadata-server, seeTroubleshoot GKE authentication issues.

Before you begin

• If you haven't already, set upauthentication. Authentication verifies your identity for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI. After installation,initialize the Google Cloud CLI by running the following command:

     gcloudinit

     If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.
  2. Set a default region and zone.

  REST

  To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

  Install the Google Cloud CLI. After installation,initialize the Google Cloud CLI by running the following command:

  gcloudinit

  If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

  For more information, seeAuthenticate for using REST in the Google Cloud authentication documentation.

Troubleshoot server codes

The following server codes are returned when you make calls to theCompute Engine metadata server. Review this section to see how to respondto each server code returned by the metadata server.

Common server codes

These server codes are frequently returned from the metadata server.

Rare server codes

These server codes, though rare, might also be returned from the metadata server.

Rate limited endpoints error codes

The following endpoints are rate limited and might return errorcodes. To handle these returned error codes, seeRetry Guidance.

Retry guidance

The metadata server routinely returns 503 and 429 error codes.To make your applications resilient, we recommend thatyou implement retry logic for applications that query the metadata server.We also suggest that you implementexponential backoff in yourscripts to account for any possible rate limiting.

Troubleshoot failed requests to the metadata server

The following are example errors that you might encounter if your VM fails toreach the metadata server:

curl: (6) Could not resolve host: metadata.google.internal

postAttribute error: Put "http://metadata.google.internal/computeMetadata/v1/instance/guest-attributes/guestInventory/ShortName": dial tcp: lookup metadata.google.internal on [::1]:53: read udp [::1]:58319->[::1]:53: read: connection refused

If your VM has lost access to the metadata server, do the following:

Troubleshoot failed requests while using a network proxy

A network proxy server prevents VM's direct access to the Internet. All queriessent from within a VM are handled by the proxy server instead.

When using an application that gets credentials from the metadata server, likean authentication token, the VM requires direct access to the metadata server.If the VM is behind a proxy, you must set theNO_PROXYconfiguration for both the IP address and Hostname.

If you don't set theNO_PROXY configuration, you might see errorswhen running Google Cloud CLI commands or querying the metadata server directlysince the calls tometadata.google.internal will be sent to theproxy without having them resolved locally on the instance itself.

The following is an example of an error that you might see:

ERROR 403 (Forbidden): Your client does not have permission to get URL

To resolve this proxy issue, add the metadata server hostname and IP address tothe environment variableNO_PROXY by doing the following:

Troubleshoot failed requests to the HTTPS metadata server endpoint

This section covers some of the errors that you might see when you arequerying the HTTPS metadata server endpoint.

The errors listed in this section are returned when you query using the cURLtool to query, however the returned error message is similar for other tools.

Incorrect client certificate

The following errors occur if you provide an incorrect value for the-Eflag.

• curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificaterequired, errno 0

• curl: (58)  unable to set private key file:

• curl: (58) could not load PEM client certificate, OpenSSL error error:02001002:system library:fopen:No such file or directory

To resolve this issue, provide the correct path to the client identitycertificate. To view the location for client identity certificates,seeClient identity certificates.

Incorrect hostname

The following error occurs if the hostname used to access the metadata serverisn't found on the certificate.

curl: (60) SSL: no alternative certificate subject name matches target host name

To resolve this issue, verify that the root URL or hostname in your query ismetadata.google.internal. For more information about the root URL forthe metadata server,seeParts of a metadata request.

Incorrect root or client certificate

You might see the following error when you query the HTTPS metadata serverendpoint:

curl: (77) error setting certificate verify locations:

This might happen in the following scenarios:

• The path for the--cacert flag might be malformed
• The root certificate might be missing from the trust store

To resolve this issue, you need to specify both theroot certificate and the client certificate when querying the https endpoint.For certificate locations, seeWhere are certificates stored.

For example, to query the boot image for a VM, run the followingquery:

user@myinst:~$curl "https://metadata.google.internal/computeMetadata/v1/instance/image" \    -E /run/google-mds-mtls/client.key \    --cacert /run/google-mds-mtls/root.crt \    -H "Metadata-Flavor: Google"

Troubleshoot incorrect header format

The metadata server performs formatting checks to ensure that headers complywith the header formatting guidelineRFC 7230 Section 3.2.If the header format fails these checks the following occurs:

• The request is accepted. However, you'll receive recommendations that youhave VMs making requests to the metadata server with incorrectly formattedheaders. Recommendations are sent once per VM.You can access these recommendations by using theGoogle Cloud CLI, or the Recommender REST API.

  After you have applied the recommendation,set the recommendation state tosucceeded.

• Starting January 20, 2024, the metadata server denies any request with anincorrectly formatted header.

The following shows examples of valid and invalid header request formats.

Not valid: contains whitespacebetween the header name and colon

Metadata-Flavor : Google

Valid: no whitespace between headername and colon, whitespace after the colon is ignored by the checker

Metadata-Flavor: Google

Valid: no whitespace in header

Metadata-Flavor:Google

For more information about making queries to metadata server, seeAccess VM metadata.

Troubleshoot failure to obtain token

You might see one of the following errors when you query the metadata server:

• ERROR: gcloud crashed (MetadataServerException): HTTP Error 401: Unauthorized

• curl: (22) The requested URL returned error: 401

These errors might arise if the service account attached to the VM is ina disabled state. To resolve these errors,enable the service account orattach different service account.