When working with Google Kubernetes Engine (GKE), issues with thekubectlcommand-line tool can prevent you from deploying applications or managingcluster resources. These problems generally fall into two categories:authentication failures, where the cluster doesn't recognize your identity, andconnectivity failures, where your tool can't reach the cluster's control plane.

Use this page to help you diagnose and resolve these issues. Find steps fortroubleshooting various authentication problems and debugging connectivityissues between thekubectl tool and your cluster's control plane. Learn tocheck that the necessary plugins are installed and configured, and review networkpolicy and firewall considerations for services like SSH and Konnectivity.

This information is important for anyone who useskubectl commands to manageapplications or cluster resources on GKE. It's particularlyimportant for Application developers and Platform admins and operators whorely onkubectl commands for their core daily tasks. For more informationabout the common roles and example tasks that we reference in Google Cloudcontent, seeCommon GKE user roles andtasks.

For related information, see the following resources:

• For more information about problems not specific to GKE, seeTroubleshooting kubectlin the Kubernetes documentation.
• For more information about how to usekubectl commands to diagnoseproblems with your clusters and workloads, seeInvestigate a cluster's state withkubectl.

Authentication and authorization errors

If you're experiencing errors related to authentication and authorization whenusing thekubectl command-line tool commands, read the following sections foradvice.

Error: 401 (Unauthorized)

When connecting to GKE clusters, you can get an authenticationand authorization error with HTTP status code401 (Unauthorized). This issuemight occur when you try to run akubectl command in your GKEcluster from a local environment. To learn more, seeIssue: Authentication and authorization errors.

Error: Insufficient authentication scopes

When you rungcloud container clusters get-credentials, you might receive thefollowing error:

ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Request had insufficient authentication scopes.

This error occurs because you are attempting to access theGKE API from a Compute Engine VM that doesn't have thecloud-platform scope.

To resolve this error, grant the missingcloud-platform scope. Forinstructions on changing the scopes on your Compute Engine VM instance, seeCreating and enabling service accounts for instancesin the Compute Engine documentation.

Error: Executable gke-gcloud-auth-plugin not found

Error messages similar to the following can occur while trying to runkubectlcommands or custom clients interacting with GKE:

Unable to connect to the server: getting credentials: exec: executable gke-gcloud-auth-plugin not foundIt looks like you are trying to use a client-go credential plugin that is not installed.To learn more about this feature, consult the documentation available at:      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-pluginsVisit cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin to install gke-gcloud-auth-plugin.

Unable to connect to the server: getting credentials: exec: fork/exec /usr/lib/google-cloud-sdk/bin/gke-gcloud-auth-plugin: no such file or directory

To resolve the issue, install thegke-gcloud-auth-plugin as described inInstall required plugins.

Error: No auth provider found

The following error occurs ifkubectl or custom Kubernetes clients have beenbuilt with Kubernetesclient-go version 1.26 or later:

no Auth Provider found for name "gcp"

To resolve this issue, complete the following steps:

1. Installgke-gcloud-auth-plugin as described inInstall required plugins.

2. Update to the latest version of the gcloud CLI:

   gcloudcomponentsupdate

3. Update thekubeconfig file:

   gcloudcontainerclustersget-credentialsCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • CONTROL_PLANE_LOCATION: the Compute Enginelocation of the control plane of yourcluster. Provide a region for regional clusters, or a zone for zonal clusters.

Error: The gcp auth plugin is deprecated, use gcloud instead

You might see the following warning message after youinstall thegke-gcloud-auth-pluginand run akubectl command against a GKE cluster:

WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.

This message appears if your client version is earlier than 1.26.

To resolve this issue, tell your client to use thegke-gcloud-auth-pluginauthentication plugin instead:

1. Open your shell login script in a text editor:

   Bash

   vi~/.bashrc

   Zsh

   vi~/.zshrc

   If you're using PowerShell, skip this step.

2. Set the following environment variable:

   Bash

   exportUSE_GKE_GCLOUD_AUTH_PLUGIN=True

   Zsh

   exportUSE_GKE_GCLOUD_AUTH_PLUGIN=True

   PowerShell

   [Environment]::SetEnvironmentVariable('USE_GKE_GCLOUD_AUTH_PLUGIN',True,'Machine')

3. Apply the variable in your environment:

   Bash

   source~/.bashrc

   Zsh

   source ~/.zshrc

   PowerShell

   Exit the terminal and open a new terminal session.

4. Update the gcloud CLI:

   gcloudcomponentsupdate

5. Authenticate to your cluster:

   gcloudcontainerclustersget-credentialsCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • CONTROL_PLANE_LOCATION: the Compute Enginelocation of the control plane of yourcluster. Provide a region for regional clusters, or a zone for zonal clusters.

Issue: Thekubectl command isn't found

If you receive a message that thekubectl command isn't found,reinstall thekubectl binary and set your$PATH environment variable:

1. Install thekubectl binary:

   gcloudcomponentsupdatekubectl

2. When the installer prompts you to modify your$PATH environmentvariable, entery to proceed. Modifying this variable lets you usekubectl commands without typing their full path.

   Alternatively, add the following line to wherever your shell storesenvironment variables, such as~/.bashrc (or~/.bash_profile inmacOS):

   exportPATH=$PATH:/usr/local/share/google/google-cloud-sdk/bin/

3. Run the following command to load your updated file. The following exampleuses.bashrc:

   source~/.bashrc

   If you are using macOS, use~/.bash_profile instead of.bashrc.

Issue:kubectl commands return "connection refused" error

Ifkubectl commands return a "connection refused" error, thenyou need to set the cluster context with the following command:

gcloudcontainerclustersget-credentialsCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• CONTROL_PLANE_LOCATION: the Compute Enginelocation of the control plane of your cluster. Provide a region for regional clusters, or a zone for zonal clusters.

If you're unsure of what to enter for the cluster name or location, use the following commandto list your clusters:

gcloudcontainerclusterslist

Error:kubectl command timed out

If you created a cluster and attempted to run akubectl command against thecluster but thekubectl command times out, you'll see an error similar to thefollowing:

• Unable to connect to the server: dial tcp IP_ADDRESS: connect: connection timed out
• Unable to connect to the server: dial tcp IP_ADDRESS: i/o timeout.

These errors indicate thatkubectl is unable to communicate with thecluster control plane.

To resolve this issue, verify and set the context where the cluster is set andensure connectivity to the cluster:

1. Go to$HOME/.kube/config or run the commandkubectl config view toverify that the config file contains the cluster context and the external IPaddress of the control plane.

2. Set the cluster credentials:

   gcloudcontainerclustersget-credentialsCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--project=PROJECT_ID

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • CONTROL_PLANE_LOCATION: the Compute Enginelocation of the control plane of yourcluster. Provide a region for regional clusters, or a zone for zonal clusters.
   • PROJECT_ID: the ID of the project that the clusterwas created in.

3. If you've enabled authorized networks in the cluster, thenensure that its list of existing authorized networks includes the outgoingIP of the machine that you are attempting to connect from. You can find yourexisting authorized networks in the console or by running thefollowing command:

   gcloudcontainerclustersdescribeCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--project=PROJECT_ID\--format"flattened(controlPlaneEndpointsConfig.ipEndpointsConfig.authorizedNetworksConfig.cidrBlocks[])"

   If the outgoing IP of the machine is not included in the list of authorizednetworks from the output of the preceding command, then complete one of thefollowing steps:

   • If you're using the console, follow the directions inCan't reach control plane of a cluster with no external endpoint.
   • If connecting from Cloud Shell, follow the directions inUsing Cloud Shell to access a cluster with external endpoint disabled.

Error:kubectl commands return failed to negotiate an api version

Ifkubectl commands return afailed to negotiate an API versionerror, then you need to ensurekubectl has authentication credentials:

gcloudauthapplication-defaultlogin

Issue:kubectllogs,attach,exec, orport-forward command stops responding

If thekubectllogs,attach,exec, orport-forward commands stopresponding, typically the API server is unable to communicate with the nodes.

First, check if your cluster has any nodes. If you've scaled down thenumber of nodes in your cluster to zero, the commands won't work. To resolvethis issue,resize your clusterto have at least one node.

If your cluster has at least one node, then check whether you are using SSH orKonnectivity proxytunnels to enable secure communication. The following sections discuss thetroubleshooting steps specific to each service:

• SSH
• Konnectivity proxy

Troubleshoot SSH issues

If you're using SSH, GKE saves an SSH public key file in yourCompute Engine projectmetadata. AllCompute Engine VMs using Google-provided images regularly check theirproject's common metadata and their instance's metadata for SSH keys to add tothe VM's list of authorized users. GKE also adds a firewall ruleto your Compute Engine network for allowing SSH access from the controlplane's IP address to each node in the cluster.

The following settings can cause issues with SSH communication:

• Your network's firewall rules don't allow for SSH accessfrom the control plane.

  All Compute Engine networks are created with a firewall rule calleddefault-allow-ssh that allows SSH access from all IP addresses (requiringa valid private key). GKE also inserts an SSH rulefor each public cluster of the formgke-CLUSTER_NAME-RANDOM_CHARACTERS-sshthat allows SSH access specifically from the cluster's control plane to thecluster's nodes.

  If neither of these rules exists, then the control planecan't open SSH tunnels.

  To verify that this is the cause of the issue, check whether yourconfiguration has these rules.

  To resolve this issue, identify the tag that's on all of the cluster's nodes,thenre-add a firewall rule allowingaccess to VMs with that tag from the IP address of the control plane.

• Your project's common metadata entry forssh-keys is full.

  If the project's metadata entry namedssh-keys is close to itsmaximum size limit,then GKE isn't able to add its own SSH key foropening SSH tunnels.

  To verify that this is the issue, check the length of the list ofssh-keys. You can see your project's metadata byrunning the following command, optionally including the--project flag:

  gcloudcomputeproject-infodescribe[--project=PROJECT_ID]

  To resolve this issue,delete some of the SSH keysthat are no longer needed.

• You have set a metadata field with the keyssh-keys on the VMs in thecluster.

  The node agent on VMs prefers per-instance SSH keys to project-wide SSH keys,so if you've set any SSH keys specifically on the cluster's nodes, then thecontrol plane's SSH key in the project metadata won't be respected by thenodes.

  To verify that this is the issue, rungcloud compute instances describeVM_NAME and look for anssh-keys field in the metadata.

  To resolve this issue,delete the per-instance SSH keysfrom the instance metadata.

Troubleshoot Konnectivity proxy issues

You can determine whether your cluster uses the Konnectivity proxyby checking for the following system Deployment:

kubectlgetdeploymentskonnectivity-agent--namespacekube-system

If your cluster uses the Konnectivity proxy, the output is similar to thefollowing:

NAME                 READY   UP-TO-DATE   AVAILABLE   AGEkonnectivity-agent   3/3     3            3           18d

After you've verified that you're using the Konnectivity proxy, make sure thatthe Konnectivity agents have the required firewall access and that your networkpolicies are set up correctly.

Allow required firewall access

Check that your network's firewall rules allow access to the following ports:

• Control plane port: On cluster creation, Konnectivity agents establishconnections to the control plane on port 8132. When you run akubectlcommand, the API server uses this connection to communicate with thecluster. Make sure you allow Egress traffic to the cluster controlplane on port 8132 (for comparison, the API server uses 443). If you haverules that deny egress access, you might need to modify the rules or createexceptions.

• kubelet port: Because Konnectivity agents are system Pods deployed on your cluster nodes, ensure that your firewall rules allow the following types of traffic:

  • Incoming traffic to your workloads at port 10250 from your Pod ranges.
  • Outgoing traffic from your Pod ranges.

  If your firewall rules don't permit this type of traffic,modify yourrules.

Adjust network policy

The Konnectivity proxy might have issues if your cluster'snetwork policy does either ofthe following:

• Blocks ingress from thekube-system namespace to theworkload namespace
• Blocks egress to the cluster control plane on port 8132

When ingress is blocked by the network policy of workload Pods, thekonnectivity-agent logs include an error message similar to thefollowing:

"error dialing backend" error="dial tcp POD_IP_ADDRESS:PORT: i/o timeout"

In the error message,POD_IP_ADDRESS is the IP addressof the workload Pod.

When egress is blocked by network policy, thekonnectivity-agent logsinclude an error message similar to the following:

"cannot connect once" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp CP_IP_ADDRESS:8132: i/o timeout

In the error,CP_IP_ADDRESS is the cluster controlplane's IP address.

These features are not required for the correct functioning of the cluster.If you prefer to keep your cluster's network locked down from alloutside access, be aware that features like these won't work.

To verify that network policy ingress or egress rules are the cause of theissue, find the network policies in the affected namespace by running thefollowing command:

kubectlgetnetworkpolicy--namespaceAFFECTED_NAMESPACE

To resolve the issue with the ingress policy, add the following to thespec.ingress field of the network policies:

ingress:-from:-namespaceSelector:matchLabels:kubernetes.io/metadata.name:kube-systempodSelector:matchLabels:k8s-app:konnectivity-agent

To resolve the issue with the egress policy, add the following to thespec.egress field of the network policies:

egress:-to:-ipBlock:cidr:CP_IP_ADDRESS/32ports:-protocol:TCPport:8132

If your network policy uses a combination of ingress and egress rules, thenconsider adjusting both.

Adjust the IP masquerade agent

The cluster control plane accepts the traffic from the Konnectivity agents if thesource IP address is in the Pod IP address ranges. If you modify theconfiguration ofip-masq-agentto masquerade the source IP address for the traffic to the cluster controlplane, the Konnectivity agents might experience connectivity errors.

To resolve the issue and to help ensure that traffic from Konnectivity agents tothe cluster control plane are not masqueraded to the node IP address, add thecontrol plane IP address to thenonMasqueradeCIDRs list in theip-masq-agentConfigMap:

nonMasqueradeCIDRs:-CONTROL_PLANE_IP_ADDRESS/32

For more information about this configuration, seeIP masqueradeagent.

Error:kubectl commands fail with no agent available error

When you runkubectl commands that need to connect from the GKEcontrol plane to a Pod—for example,kubectl exec,kubectl logs, orkubectlport-forward—the command might fail with error messages similar to thefollowing:

Error from server: error dialing backend: No agent available

failed to call webhook: Post "https://WEBHOOK_SERVICE.WEBHOOK_NAMESPACE.svc:PORT/PATH?timeout=10s": No agent available

v1beta1.metrics.k8s.io failed with: failing or missing response from https://NODE_IP:10250/apis/metrics.k8s.io/v1beta1: Get "https://NODE_IP:10250/apis/metrics.k8s.io/v1beta1": No agent available

These errors indicate a problem with Konnectivity, the secure communicationtunnel between the GKE control plane and your cluster's nodes. Inparticular, it means thekonnectivity-server on the control plane cannotconnect to any healthykonnectivity-agent Pods in thekube-system namespace.

To resolve this issue, try the following solutions:

1. Verify the health of thekonnectivity-agent Pods:

   1. Check if thekonnectivity-agent Pods are running:

      kubectlgetpods-nkube-system-lk8s-app=konnectivity-agent

      The output is similar to the following:

      NAME                                   READY   STATUS    RESTARTS  AGEkonnectivity-agent-abc123def4-xsy1a    2/2     Running   0         31dkonnectivity-agent-abc123def4-yza2b    2/2     Running   0         31dkonnectivity-agent-abc123def4-zxb3c    2/2     Running   0         31d

      Review the value in theStatus column. If the Pods have a status ofRunning, review logs for connection issues. Otherwise, investigate whythe Pods aren't running.

   2. Review logs for connection issues. If the Pods have a status ofRunning, check logs for connection issues. Because thekubectl logscommand depends on Konnectivity, use Logs Explorer in theGoogle Cloud console:

      1. In the Google Cloud console, go toLogs Explorer.

         Go to Logs Explorer

      2. In the query pane, enter the following query.

         resource.type="k8s_container"resource.labels.cluster_name="CLUSTER_NAME"resource.labels.namespace_name="kube-system"labels."k8s-pod/k8s-app"="konnectivity-agent"resource.labels.container_name="konnectivity-agent"

         ReplaceCLUSTER_NAME with the name of yourcluster.

      3. ClickRun query.

      4. Review the output. When you review thekonnectivity-agent logs,look for errors indicating why the agent can't connect.Authentication or permission errors often point to a misconfiguredwebhook blocking token reviews. "Connection refused" or "timeout"errors typically mean a firewall rule or Network Policy is blockingtraffic to the control plane on TCP port 8132, or is blocking trafficbetween the Konnectivity agent and other nodes. Certificate errorssuggest that a firewall or proxy is inspecting and interfering withthe encrypted TLS traffic.

   3. Investigate why the Pods aren't running. If Pods have a status ofPending or another non-running state, investigate the cause. Thekonnectivity-agent runs as a Deployment, not a DaemonSet. Because itruns as a Deployment, the agent Pods only need to run on a subset ofnodes. However, if that specific subset of nodes is unavailable, theentire service can fail.

      Common causes of a non-running Pod include the following:

      • Custom node taints that prevent a Pod from being scheduled.
      • Insufficient node resources (CPU or memory).
      • Restrictive Binary Authorization policies that blockGKE system images.

      To get more details about why a specific Pod isn't running, use thekubectl describe command:

      kubectldescribepodPOD_NAME-nkube-system

      ReplacePOD_NAME with the name of the Pod that isn'trunning.

2. Investigate your admission webhooks to ensure none are blockingTokenReview API requests. Thekonnectivity-agent relies on serviceaccount tokens, so interference with token reviews can prevent agents fromconnecting. If a webhook is the cause, Konnectivitycannot recover untilthe faulty webhook is removed or repaired.

3. Ensure your firewall rules allow TCP egress traffic from yourGKE nodes to the control plane's IP address on port 8132.This connection is required for thekonnectivity-agent to reach theKonnectivity service. For more information, seeAllow required firewallaccess.

4. Make sure that there are noNetwork Policy rulesthat restrict essential Konnectivity traffic. Network Policy rules shouldallow both intra-cluster traffic (Pod-to-Pod) within thekube-systemnamespace and egress traffic from thekonnectivity-agent Pods to theGKE control plane.

Troubleshootkubectl client-side throttling

Symptom:

You might experience client-side throttling when usingkubectl, with errormessages that contain the following:...Throttling request took 1.002582473s, request: GET:... orWaited for 1.183040416s due to client-side throttling, not priority and fairness, request: GET: ....

Cause:

When you run akubectl command, it first caches a list of resources from theAPI server. This list corresponds to the Custom Resource Definitions (CRDs)available from the API server.

When a GKE cluster has a large number of CRDs, for example 300 or more,kubectl sends a large number of requests to the API server to build or refreshits cache. To avoid overloading the API server,kubectl throttles itself.

Resolution:

These types of throttling error messages are expected behavior and are caused byclient-side throttling on thekubectl side. This issue should resolve on its own.

What's next

• If you can't find a solution to your problem in the documentation, seeGet support for further help,including advice on the following topics:

  • Opening a support case by contactingCloud Customer Care.
  • Getting support from the community byasking questions on StackOverflow and using thegoogle-kubernetes-engine tag to search for similarissues. You can also join the#kubernetes-engine Slack channel for more community support.
  • Opening bugs or feature requests by using thepublic issue tracker.