Resolving sidecar proxy/webhook issues in Cloud Service Mesh

This section explains common Cloud Service Mesh problems and how to resolvethem. If you need additional assistance, seeGetting support.

Cloud Service Mesh contains twowebhooks:

• The validating webhook ensures applied Istio configuration is valid.
• The mutating webhook sets automatic sidecar injection on new pods.

A configuration issue in one of these webhooks might cause new pods to failstart up, orkubectl apply generating error messages.

Sidecar injection problems

If you have provisioned managed Cloud Service Mesh, thencontact support.

Sidecar injection is not working correctly if you see any of the following:

• pods that are scheduling without sidecars
• pods that should have sidecars injected never appear when usingkubectl get pods, but the corresponding replica set fromkubectl get replicaset exists.

Use the following steps to troubleshoot sidecar injection problems.

1. Verify that your namespace or pod has the correct injection label.

   If you are running single-revision Istio (the default), verify that yournamespace or pod spec have the istio-injection=enabled label.

   If you are running multiple-revision Istio (for zero-downtime migrations,multiple control planes, etc), verify that your namespace or pod spec havethe appropriateistio.io/rev=REVISION label, whereREVISION is the Cloud Service Mesh revision number onistiodthat corresponds with your selected Cloud Service Mesh version. For moreinformation about revision labels, seeInjecting sidecar proxies.

2. Verify that your istio sidecar injection webhook is present and has a CA bundle.

   The sidecar injector webhook (which is used for automatic sidecar injection)requires a CA bundle to establish secure connections with the API server andistiod. This CA bundle is patched into the configuration byistiod, butcan sometimes be overwritten (for example, if you reapply the webhookconfiguration).

   You can verify the presence of the CA bundle using the following command. Thecommand includesistio-sidecar-injector-asm-1282-4, which isspecific to this version of Cloud Service Mesh. Ensure you use your actualrevision if it differs.

   kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io istio-sidecar-injector-asm-1282-4 -o=jsonpath='{.webhooks[0].clientConfig.caBundle}'

   If the output is not empty, the CA bundle is configured. If the CA bundle ismissing, restartistiod to cause it to rescan the webhook andreinstall the CA bundle.

3. Check for sidecar injection failures.

   If you have injection enabled, but are not seeing pods scheduling, check thestatus of the next higher level of abstraction. For example, if you arerunning a deployment but no pods are scheduling, check the status of thecorresponding replica sets using the following command:

   kubectl -n my-namespace describe replicaset your-replicaset-name

   If the replica set is present, check the events log at the bottom of thedescription for errors.

   The errorService Mesh injection is not yet ready. Please try again soon. indicates that Cloud Service Mesh is still applying user configurations. Therefore, new workloads cannot be deployed at this time. This is expected for newly provisioned clusters. This is usually a brief process, resolving itself within minutes through automatic retries. Wait a short while for the configuration process to complete.

   If the error relates to any other sidecar injection issue, check theistiod logs for an indication of what is causing the error.

4. If the problem persists, the issue might be any of the following:

   • Bad configuration passed to the injector
   • Firewall configuration problems
   • A problem in the Istio code itself

   SeeTroubleshooting Istiofor additional diagnostic steps.

Envoy proxies don't receive configuration fromistiod

There are several issues that can prevent proxies from receiving configurationfromistiod.

1. istiod will not push configuration to the envoy proxies if it has problems,such as an RBAC issue preventing it from reading its configuration resource.

2. Discovery address is incorrect ('no healthy upstream' errors)

3. The discovery address provided to the sidecar injector being incorrect. Ifyou see logs that mentiongRPC config stream closed, no healthy upstream,check that the discovery address in the meshProxyConfigis correct and points to youristiod service.

4. Invalid configuration being pushed to the proxy. In this case, configurationis successfully pushed to the proxy, but the configuration is invalid. You willsee repeating messages similar to the following:

   Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected

   In this example,cds is the Cluster Discovery Service (which reports 1update pushed fromistiod), andlds is the Listener Discovery Service(which reports 1 update rejected fromistiod). Often you will see anearlier error message that explains the reason for the rejection, whichusually starts with a warning about envoy configuration or similar.

   To fix the issue, investigate the cause of the rejected configuration. Onecommon cause is badEnvoyFilter resources. If no reason is obvious,submit a bug report with a configuration dump of the proxy.

Pod creation fails

If you observe that pods are not being created successfully, look for errormessages that might give clues to the root problem, using the following command:

kubectl describe replicasetYOUR_REPLICA_SET

Common webhook error messages

Error messages output by thekubectl apply command can provide a hint abouttheir root cause. See the following table for common error messages, theircauses and potential resolutions.