Ensure control plane stability when using webhooks

Admission webhooks, orwebhooks in Kubernetes, are a type ofadmissioncontroller,which can be used in Kubernetes clusters to validate or mutate requests to thecontrol plane prior to a request being persisted. It is common for third-partyapplications to use webhooks that operate on system-critical resources andnamespaces. Incorrectly configured webhooks can impact control planeperformance and reliability. For example, an incorrectly configured webhookcreated by a third-party application could prevent GKE from creating andmodifying resources in the managedkube-system namespace, which could degradethe functionality of the cluster.

Google Kubernetes Engine (GKE) monitors your clusters and uses theRecommender service to deliver guidancefor how you can optimize your usage of the platform. To help you ensure thatyour cluster remains stable and performant, see recommendations fromGKE for the following scenarios:

  • Webhooks that operate but have no endpoints available.
  • Webhooks that are considered unsafe as they operate on system critical resources and namespaces.

With this guidance, you can see instructions for how to check your potentiallymisconfigured webhooks and update them, if necessary.

To learn more about how to manage insights and recommendations fromRecommenders, seeOptimize your usage of GKE with insights andrecommendations.

Identify misconfigured webhooks that could affect your cluster

To get insights identifying webhooks that could affect your cluster'sperformance and stability, follow the instructions toview insights andrecommendations.You can get insights in the following ways:

  • Use the Google Cloud console.
  • Use the Google Cloud CLI, or the Recommender API, filtering with thesubtypesK8S_ADMISSION_WEBHOOK_UNSAFE andK8S_ADMISSION_WEBHOOK_UNAVAILABLE.

After you identify the webhooks via the insights, follow the instructions totroubleshoot the detected webhooks.

When GKE detects misconfigured webhooks

GKE generates an insight and recommendation if either of the following criteria are true for a cluster:

Troubleshoot the detected webhooks

The following sections have instructions for you to troubleshoot the webhooksthat GKE detected as potentially misconfigured.

After you implement the instructions and the webhooks are correctly configured,the recommendation is resolved within 24 hours and no longer appears in theconsole.

If you do not want to implement the recommendation, you candismiss it.

Webhooks reporting no available endpoints

If a webhook is reporting that it has no available endpoints, the Service thatis backing the webhook endpoint has one or more Pods which are not running. Tomake the webhook endpoints available, follow the instructions to find andtroubleshoot the Pods of the Service that is backing this webhook endpoint:

  1. View insights and recommendations,choosing one insight at a time to troubleshoot. GKE generatesone insight per cluster, and this insight lists one or more webhooks with abroken endpoint that must be investigated. For each of these webhooks, theinsight also states the Service name, what endpoint is broken, and thelast time that the endpoint was called.

  2. Find the serving Pods for the Service associated with the webhook:

    Console

    From the insight's sidebar panel, see the table of misconfigured webhooks.Click on the name of the Service.

    kubectl

    Run the following command to describe the Service:

    kubectldescribesvcSERVICE_NAME-nSERVICE_NAMESPACE

    ReplaceSERVICE_NAME andSERVICE_NAMESPACE with the name and namespace ofthe service, respectively.

    If you cannot find the Service name listed in the webhook, the unavailableendpoint might be caused by a mismatch between the name listed in theconfiguration and the actual name of the Service. To fix the endpointavailability, update the Service name in the webhook configuration to matchthe correct Service object.

  3. Inspect the serving Pods for this Service:

    Console

    UnderServing Pods in theService details, see the list of Podsbacking this Service.

    kubectl

    Identify which Pods are not running by listing the Deployment or Pods:

    kubectlgetdeployment-nSERVICE_NAMESPACE

    Or, run this command:

    kubectlgetpods-nSERVICE_NAMESPACE-owide

    For any Pods that are not running, inspect the Pod logs to see why the Podis not running. For instructions on common issues with Pods, seeTroubleshoot issues with deployed workloads.

Webhooks that are considered unsafe

If a webhook is intercepting any resources in system-managed namespaces, orcertain types of resources, GKE considers thisunsafe and recommends that you update the webhooks to avoid intercepting theseresources.

  1. Follow the instructions toview insights and recommendations,choosing one insight at a time to troubleshoot. GKE onlygenerates one insight per cluster, and this insight lists one or more webhookconfigurations, each of which lists one or more webhooks. For each webhookconfiguration listed, the insight states the reason why the configurationwas flagged.

  2. Inspect the webhook configuration:

    Console

    From the insight's sidebar panel, see the table. In each row is the nameof the webhook configuration, and the reason why this configuration wasflagged.

    To inspect each configuration, click the name to navigate to thisconfiguration in the GKEObject Browserdashboard.

    kubectl

    Run the followingkubectl command to get the webhook configuration,replacingCONFIGURATION_NAME with the name of thewebhook configuration:

    kubectl get validatingwebhookconfigurationsCONFIGURATION_NAME -o yaml

    If this command doesn't return anything, run the command again, replacingvalidatingwebhookconfigurations withmutatingwebhookconfigurations.

    In thewebhooks section, there are one or more webhooks listed.

  3. Edit the configuration, depending on the reason the webhook was flagged:

    Exclude kube-system and kube-node-lease namespaces

    A webhook is flagged ifscope is*. Or, a webhook is flagged if scope isNamespaced and either of the following conditions are true:

    • Theoperator condition isNotIn andvalues omitskube-system andkube-node-lease, as in the following example:

      webhooks:-admissionReviewVersions:...namespaceSelector:matchExpressions:-key:kubernetes.io/metadata.nameoperator:NotInvalues:-blue-systemobjectSelector:{}rules:-apiGroups:...scope:'*'sideEffects:NonetimeoutSeconds:3

      Ensure that you setscope toNamespaced, not*, so that thewebhook only operates in specific namespaces. Also ensure that if theoperator isNotIn, you includekube-system andkube-node-leaseinvalues (in this example, withblue-system).

    • Theoperator condition isIn andvalues includeskube-system andkube-node-lease, as in the following example:

      namespaceSelector:matchExpressions:-key:kubernetes.io/metadata.nameoperator:Invalues:-blue-system-kube-system-kube-node-lease

      Ensure that you setscope toNamespaced, not*, so that the webhookonly operates in specific namespaces. Ensure that ifoperator isIn,you don't includekube-system andkube-node-lease invalues. In thisexample, onlyblue-system should be invalues as theoperator isIn.

    Exclude matched resources

    A webhook is also flagged ifnodes,tokenreviews,subjectaccessreviews,orcertificatesigningrequests are listed under resources, as in thefollowing example:

    -admissionReviewVersions:...resources:-'pods'-'nodes'-'tokenreviews'-'subjectaccessreviews'-'certificatesigningrequests'scope:'*'sideEffects:NonetimeoutSeconds:3

    Removenodes,tokenreviews,subjectaccessreviews, andcertificatesigningrequests from the resource section. You can keeppods inresources.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-10-24 UTC.