This document explains how to troubleshoot common errors related toAuthenticate workloads to other workloads over mTLS.

Before you begin

• If you haven't already, set upauthentication. Authentication verifies your identity for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

  1. Install the Google Cloud CLI. After installation,initialize the Google Cloud CLI by running the following command:

     gcloudinit

     If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.
  2. Set a default region and zone.

The generated credentials directory doesn't exist

If you get an error that the/var/run/secrets/workload-spiffe-credentialsdirectory doesn't exist, do the following:

1. Ensure that your VM supports workload to workload authentication by runningthe following command from inside the VM.

   curl  "http://metadata.google.internal/computeMetadata/v1/instance/gce-workload-certificates/config-status" -H "Metadata-Flavor: Google"

   1. If the response is anHTTP 404 error code with the following errormessage, then this VM doesn't support this feature.

      The requested URL /computeMetadata/v1/instance/gce-workload-certificates/config-statuswas not found on this server.  That's all we know.

      To resolve, create a new VM that supports workload to workloadauthentication, using one of the following methods:

      • Enable managed workload identities for a MIG.
      • Enable managed workload identities for individual VMs.

   2. If the response is anHTTP 404 error code with the error messageworkload certificate feature not enabled, then the VM supports managedworkload identities, but the feature isn't enabled. To enable the featureon the VM, seeEnable managed workload identities on existing VMs.

2. Ensure the VM is running a guest OS with Compute Engine guest agentversion 20231103.01 or newer. Use the gcloud CLI toview the serial-port outputto determine the current Compute Engine guest agent version:

   gcloudcomputeinstancesget-serial-port-outputVM_NAME|grep"GCE Agent Started"

   ReplaceVM_NAME with the name of the VM.

   To update the Compute Engine guest agent, seeUpdating the guest environment.

3. Check the service logs to verify that thegce-workload-cert-refresh.timersystemd timer was able to successfully fetch the workload credentials and thetrust bundle.

   Note: For guest agent version20250901.00 and later, the workload refreshservice is included with the guest agent's core plugin, not a separateservice. For more information about this version, see theGuest environmentrelease notes.

   # View timer logs to see when the gce-workload-cert-refresh.timer last ranjournalctl -u gce-workload-cert-refresh.timer# View service logs from gce-workload-cert-refresh.servicejournalctl -u gce-workload-cert-refresh.service

The generated credentials directory contains only the config_status file

The generated credentials directory,/var/run/secrets/workload-spiffe-credentials, might contain only theconfig_status for a variety of reasons. Use the following steps totroubleshoot this issue.

1. Check the contents of theconfig_status file to ensure that themanaged workload identities feature is enabled. If the feature is notenabled using the appropriate VM metadata, the log file contains theerror messageworkload certificate feature not enabled.

   To resolve this issue, create a new VM that supports workload to workloadauthentication, using one of the following methods:

   • Enable managed workload identities for a MIG.
   • Enable managed workload identities for individual VMs.

2. Check the contents of theconfig_status file to ensure that there are noerrors due to missing attribute values or invalid configuration for thecertificate issuance or the trust config. If sucherrors exist, update the configuration values by following the steps inUpdate certificate issuance and trust config.

3. Ensure that the correct permissions were granted to the managed workloadidentities in the workload identity pool for accessing the subordinate CApools. Use the following command:

   gcloud privateca pools get-iam-policySUBORDINATE_CA_POOL_ID \   --location=SUBORDINATE_CA_POOL_REGION

   Replace the following:

   • SUBORDINATE_CA_POOL_ID: the ID for the subordinate CApool.
   • SUBORDINATE_CA_POOL_REGION: the region of the subordinateCA pool.

   The output of this command should contain the following:

   bindings:-members:-principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/*-role:roles/privateca.poolReader-members:-principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/*role:roles/privateca.workloadCertificateRequester

   In the previous example:

   • PROJECT_NUMBER is the project number of your project.
   • POOL_ID is the ID of the workload identity pool.

   If you don't see output similar to the preceding example, grant the requiredpermissions as described inAuthorize managed workload identities to request certificates from the CA pool.

4. If theconfig_status file contains no error messages, then check the valueofiam.googleapis.com/workload-identity within the file.The value shouldmatch the following:

   spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog/ns/NAMESPACE_ID/sa/MANAGED_IDENTITY_ID

   In the previous example:

   • PROJECT_NUMBER is the project number for the project thatcontains the managed workload identity pool.
   • POOL_ID is the ID of the workload identity pool.
   • NAMESPACE_ID is the ID of the namespace in the workloadidentity pool.
   • MANAGED_IDENTITY_ID is the ID of the managed workloadidentity.

   If the value foriam.googleapis.com/workload-identity is incorrect, thenyou must create a new VM with the correct value because the managed identityvalue can only be updated during VM creation.

5. If theconfig_status file contains no error messages, then ensure that thetrust config contains a valid entry for the SPIFFE trust domainPOOL_ID.global.PROJECT_NUMBER.workload.id.goog,which corresponds to the SPIFFE trust domain on the managed identityassigned to the VM. For more information, seeDefine the trust config.

6. If theconfig_status file contains any error messages with the error codeINTERNAL_ERROR, reach out to Cloud Customer Care or your Google Cloud contactwith the error message.

Querying metadata server endpoints returns a 404 error

If you get a404 response when querying theworkload-identities ortrust-anchors endpoint, then ensure that the VM supports the managed workload identities by runningthe following command from inside the VM:

curl  "http://metadata.google.internal/computeMetadata/v1/instance/gce-workload-certificates/config-status" -H "Metadata-Flavor: Google"

• If the response is an HTTP404 error code with the following error message:

    The requested URL /computeMetadata/v1/instance/gce-workload-certificates/config-status  was not found on this server.  That's all we know.

  The VM doesn't support managed workload identities. To resolve the issue, do one ofthe following:

  • Enable managed workload identities for a MIG.
  • Enable managed workload identities for individual VMs.

• If the response is an HTTP404 error code with the error messageworkload certificate feature not enabled, then this VM supportsmanaged workload identities, but the feature is not enabled.Create a new VM with the feature enabled,or create a new instance template and managed instance group.

• Ensure that the correct permissions were granted to the workload identity poolfor accessing the subordinate CA pools by running the following command:

  gcloud privateca pools get-iam-policySUBORDINATE_CA_POOL_ID \  --location=SUBORDINATE_CA_POOL_REGION

  Replace the following:

  • SUBORDINATE_CA_POOL_ID: the ID for the subordinate CA pool.
  • SUBORDINATE_CA_POOL_REGION: the region of the subordinateCA pool.

  The output of this command should contain the following, wherePROJECT_NUMBER is the project number of your project andPOOL_ID is the ID of the workload identity pool.

  bindings:- members:- principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/*- role: roles/privateca.poolReader- members:- principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/*- role: roles/privateca.workloadCertificateRequester

  If your output doesn't contain these values, grant the correct permissions,as described inAuthorize managed workload identities to request certificates from the CA pool.

• Ensure that theiam.googleapis.com/workload-identity value is correct andmatches the following:

  spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog/ns/NAMESPACE_ID/sa/MANAGED_IDENTITY_ID

  If the value doesn't match, then you must create a new VM because the managedidentity value can't be updated after creating the VM.

• Ensure that thetrust configcontains a valid entry for the SPIFFE trust domainPOOL_ID.global.PROJECT_NUMBER.workload.id.goog,which corresponds to the SPIFFE trust domain on the managed identity assignedto the VM.