This page shows you how to disable the insecure kubelet read-only port inGoogle Kubernetes Engine (GKE) clusters to reduce the risk of unauthorized access tothe kubelet, and how to migrate applications to a more secure port.

In Kubernetes clusters, including GKE, thekubelet processrunning on nodes serves a read-only API using the insecure port10255.Kubernetes doesn't perform any authentication or authorization checks on thisport. The kubelet serves the same endpoints on the more secure, authenticatedport10250.

Disable the kubelet read-only port and switch any workloads that use port10255 to use the more secure port10250 instead.

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Ensure that you have an existing Autopilot or Standardcluster. To create a new cluster, seeCreate an Autopilot cluster.

Requirements

• You can only disable the insecure kubelet read-only port in GKEversion 1.26.4-gke.500 or later.

Check for insecure port usage and migrate applications

Before you disable the insecure read-only port, migrate any of your runningapplications that use the port to the more secure read-only port. Workloads thatmight need migration include custom metrics pipelines and workloads thataccess kubelet endpoints.

• For workloads that need access to the information served by the kubelet APIon the node, like metrics, use port10250.
• For workloads that get Kubernetes information using the kubelet API on thenode, like listing Pods on the node, use the Kubernetes API instead.

Check whether applications use the insecure kubelet read-only port

This section shows you how to check for insecure port usage in your cluster.

Check for port usage on Autopilot mode

To check for port usage in an Autopilot cluster, ensure that you haveat least one workload that isn't a DaemonSet running in the cluster. If youperform the following steps on an empty Autopilot cluster, the resultsmight be invalid.

1. Save the following manifest asread-only-port-metrics.yaml:

   # Create a namespace for the DaemonSet that checks for port usage.apiVersion:v1kind:Namespacemetadata:name:node-metrics-printer-namespace---# Grant access to read node metrics in the cluster.apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRolemetadata:name:node-metrics-printer-rolerules:-apiGroups:-""resources:-nodes/metricsverbs:-get---apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRoleBindingmetadata:name:node-metrics-printer-bindingroleRef:apiGroup:rbac.authorization.k8s.iokind:ClusterRolename:node-metrics-printer-role# Bind the ClusterRole to the ServiceAccount that the DaemonSet will use.subjects:-kind:ServiceAccountname:node-metrics-printer-sanamespace:node-metrics-printer-namespace---# Create a ServiceAccount for the DaemonSet.apiVersion:v1kind:ServiceAccountmetadata:name:node-metrics-printer-sanamespace:node-metrics-printer-namespace---apiVersion:apps/v1kind:DaemonSetmetadata:name:node-metrics-printernamespace:node-metrics-printer-namespacespec:selector:matchLabels:app:node-metrics-printertemplate:metadata:labels:app:node-metrics-printerspec:# Assign the ServiceAccount to the DaemonSet.serviceAccountName:node-metrics-printer-sacontainers:-name:metrics-printerimage:us-docker.pkg.dev/cloud-builders/ga/v1/curl:latestcommand:["sh","-c"]# Call the /metrics endpoint using the insecure kubelet read-only# port.args:-'whiletrue;docurl-s--cacert"${CA_CERT}"-H"Authorization:Bearer$(cat${TOKEN_FILE})""https://${NODE_ADDRESS}:10250/metrics"|grepkubelet_http_requests_total;sleep20;done'env:# Provide credentials and the IP address for the command.-name:CA_CERTvalue:/var/run/secrets/kubernetes.io/serviceaccount/ca.crt-name:TOKEN_FILEvalue:/var/run/secrets/kubernetes.io/serviceaccount/token-name:NODE_ADDRESSvalueFrom:fieldRef:fieldPath:status.hostIP

   This manifest does the following:

   1. Creates a namespace and sets up RBAC roles to allow reading nodemetrics.
   2. Deploys a DaemonSet that checks the kubelet metrics for theinsecure read-only port.

2. Deploy the manifest:

   kubectlcreate-fread-only-port-metrics.yaml

3. Check the DaemonSet logs:

   kubectllogs--namespace=node-metrics-printer-namespace\--all-containers--prefix\--selector=app=node-metrics-printer

   If the output has any results that contain the stringserver_type=readonly,an application is using the insecure read-only port.

Check for port usage on Standard mode

Run the following command on at least one node in every node pool in yourcluster:

kubectlget--raw/api/v1/nodes/NODE_NAME/proxy/metrics|grephttp_requests_total|grepreadonly

ReplaceNODE_NAME with the name of the node.

If the output contains entries with theserver_type="readonly" string, then the following scenarios can occur:

• Workloads on the node use the insecure kubelet read-only port.
• After disabling the insecure port, the command still returns theserver_type="readonly" string. This is because thekubelet_http_requests_total metric represents the cumulative number of HTTP requests received by the kubelet server since its last restart.This number is not reset when the insecure port is disabled. This number is reset after GKE restarts the kubelet server, such as during a node upgrade. To learn more, seeKubernetes Metrics Reference.

If the output is empty, no workloads on that node use the insecureread-only port.

Identify the workloads that are using the insecure kubelet read-only port

To identify the workloads that are using the insecure port, check the workload's configuration files such as ConfigMaps and Pods.

Run the following commands:

kubectlgetpods--all-namespaces-oyaml|grep10255kubectlgetconfigmaps--all-namespaces-oyaml|grep10255

If the output of the command is not empty, use the following script to identify the names of the ConfigMaps or Pods that are using the insecure port:

# This function checks if a Kubernetes resource is using the insecure port 10255.## Arguments:#  $1 - Resource type (e.g., pod, configmap, )#  $2 - Resource name#  $3 - Namespace## Output:#  Prints a message indicating whether the resource is using the insecure port.isUsingInsecurePort(){resource_type=$1resource_name=$2namespace=$3config=$(kubectlget$resource_type$resource_name-n$namespace-oyaml)# Check if kubectl output is emptyif[[-z"$config"]];thenecho"No configuration file detected for$resource_type:$resource_name (Namespace:$namespace)"returnfiifecho"$config"|grep-q"10255";thenecho"Warning: The configuration file ($resource_type:$namespace/$resource_name) is using insecure port 10255. It is recommended to migrate to port 10250 for enhanced security."elseecho"Info: The configuration file ($resource_type:$namespace/$resource_name) is not using insecure port 10255."fi}# Get the list of ConfigMaps with their namespacesconfigmaps=$(kubectlgetconfigmaps-A-ocustom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name|tail-n+2|awk'{print $1"/"$2}')# Iterate over each ConfigMapforconfigmapin$configmaps;donamespace=$(echo$configmap|cut-d/-f1)configmap_name=$(echo$configmap|cut-d/-f2)isUsingInsecurePort"configmap""$configmap_name""$namespace"done# Get the list of Pods with their namespacespods=$(kubectlgetpods-A-ocustom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name|tail-n+2|awk'{print $1"/"$2}')# Iterate over each Podforpodin$pods;donamespace=$(echo$pod|cut-d/-f1)pod_name=$(echo$pod|cut-d/-f2)isUsingInsecurePort"pod""$pod_name""$namespace"done

Once you've identified the relevant workloads, migrate them to use the secure port 10250 by completing the steps in the following section.

Migrate from the insecure kubelet read-only port

Typically, migrating an application to the secure port involves the followingsteps:

1. Update URLs or endpoints that refer to the insecure read-only port to usethe secure read-only port instead. Forexample, changehttp://203.0.113.104:10255 tohttp://203.0.113.104:10250.

2. Set the certificate authority (CA) certificate of the HTTP client to thecluster CA certificate. To find this certificate, run the followingcommand:

   gcloudcontainerclustersdescribeCLUSTER_NAME\--location=LOCATION\--format="value(masterAuth.clusterCaCertificate)"

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • LOCATION: the location of your cluster.

The authenticated port10250 requires that you grant appropriate RBACroles to the subject to access the specific resources. For details, in theKubernetes documentation, seekubelet authorization.

If your workload uses the/pods endpoint on the insecure kubelet read-onlyport, you need to grant thenodes/proxy RBAC permission to access the endpointon the secure kubelet port.nodes/proxy is a powerful permission that youcan't grant in GKE Autopilot clusters and that youshouldn't grant in GKE Standard clusters. Use theKubernetes API with afieldSelectorfor the node name instead.

If you use third-party applications that depend on the insecure kubeletread-only port, check with the application vendor for instructions to migrateto secure port10250.

Example migration

Consider a Pod that queries metrics from the insecure kubelet read-only port.

apiVersion:v1kind:Podmetadata:name:kubelet-readonly-examplespec:restartPolicy:Nevercontainers:-name:kubelet-readonly-exampleimage:us-docker.pkg.dev/cloud-builders/ga/v1/curl:latestcommand:-curl-http://${NODE_ADDRESS}:10255/metricsenv:-name:NODE_ADDRESSvalueFrom:fieldRef:fieldPath:status.hostIP

This application does the following:

• Uses thedefault ServiceAccount in thedefault namespace
• Runs thecurl command against the/metrics endpoint on the node.

To update this Pod to use the secure port10250, do the following steps:

1. Create a ClusterRole with access to get node metrics:

   apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRolemetadata:name:curl-authenticated-rolerules:# Grant access to read node metrics in the cluster.-apiGroups:-""resources:-nodes/metricsverbs:-get

2. Bind the ClusterRole to your application's identity:

   apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRoleBindingmetadata:name:curl-authenticated-role-bindingroleRef:apiGroup:rbac.authorization.k8s.iokind:ClusterRolename:curl-authenticated-role# Bind the ClusterRole to the default ServiceAccount in the default# namespace.subjects:-kind:ServiceAccountname:defaultnamespace:default

3. Update thecurl command to use the secure port endpoint with thecorresponding authorization headers:

   apiVersion:v1kind:Podmetadata:name:kubelet-authenticated-examplespec:restartPolicy:Nevercontainers:-name:kubelet-readonly-exampleimage:us-docker.pkg.dev/cloud-builders/ga/v1/curl:latestenv:-name:NODE_ADDRESSvalueFrom:fieldRef:fieldPath:status.hostIP# Update the command to send a request with the ServiceAccount# credentials in the header.command:-sh--c-'curl-s--cacert/var/run/secrets/kubernetes.io/serviceaccount/ca.crt-H"Authorization:Bearer$(cat/var/run/secrets/kubernetes.io/serviceaccount/token)"https://${NODE_ADDRESS}:10250/metrics'

Modify VPC firewall rules

If you update workloads to use port10250, create firewall rules so that Podsin the cluster can reach the port in your node IP address ranges. The firewallrules should do the following:

• Allow incoming traffic to TCP port10250 on your node IP address rangesfrom internal Pod IP address ranges
• Deny incoming traffic to TCP port10250 on your node IP address ranges fromthe public internet.

You can use the followingdefault GKE firewall rulesas a template for the parameters to specify in your new rules:

• gke-[cluster-name]-[cluster-hash]-inkubelet
• gke-[cluster-name]-[cluster-hash]-exkubelet

Disable the insecure read-only port on Autopilot clusters

You can disable the insecure kubelet read-only port for new and existingAutopilot clusters.

To disable the insecure kubelet read-only port on an Autopilotcluster, use the--no-autoprovisioning-enable-insecure-kubelet-readonly-port flag, like in thefollowing command. All new and existing nodes in the cluster stop using theport.

gcloudcontainerclustersupdateCLUSTER_NAME\--location=LOCATION\--no-autoprovisioning-enable-insecure-kubelet-readonly-port

You can also use the--no-autoprovisioning-enable-insecure-kubelet-readonly-port flag when youcreate a new cluster by using thegcloud container clusters create-auto command.

Disable the insecure read-only port on Standard clusters

You can disable the insecure kubelet read-only port for entire Standardclusters or for individual node pools. We recommend that you disable the portfor the entire cluster.

If you use node auto-provisioning, automatically provisioned node pools inheritthe port setting that you specify at the cluster level. You can optionallyspecify a different setting for auto-provisioned node pools, but we recommendthat you disable the port across all nodes in your cluster.

You can also use anode system configuration fileto declaratively disable the insecure kubelet read-only port. If you use thisfile, you can't use the commands in the following sections to control thekubelet setting.

Disable the insecure read-only port on existing Standard clusters

To disable the insecure kubelet read-only port on an existing Standardcluster, use the--no-enable-insecure-kubelet-readonly-port flag like in thefollowing command. Anynew node pools won't use the insecure port.GKE doesn't update existing node pools automatically.

gcloudcontainerclustersupdateCLUSTER_NAME\--location=LOCATION\--no-enable-insecure-kubelet-readonly-port

Replace the following:

• CLUSTER_NAME: the name of your existing Standardcluster.
• LOCATION: the location of your existing Standardcluster.

You can also use the--no-autoprovisioning-enable-insecure-kubelet-readonly-port flag when youcreate a new cluster by using thegcloud container clusters create command.

Disable the insecure read-only port on Standard node pools

We recommend that you set the read-only port setting at the cluster level in allcases. If you disabled the read-only port on an existing cluster that alreadyhad running node pools, use the following command to disable the port on thosenode pools.

gcloudcontainernode-poolsupdateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--location=LOCATION\--no-enable-insecure-kubelet-readonly-port

gcloudcontainerclustersdescribeCLUSTER_NAME\--location=LOCATION\--flatten=nodePoolAutoConfig\--format="value(nodeKubeletConfig)"

insecureKubeletReadonlyPortEnabled: false

gcloudcontainerclustersdescribeCLUSTER_NAME\--location=LOCATION\--flatten=nodePoolDefaults.nodeConfigDefaults\--format="value(nodeKubeletConfig)"

insecureKubeletReadonlyPortEnabled: false

If the output of this command is blank, the insecurekubelet read-only portmight still be enabled. To disable the port, run the command in theDisable the insecure read-only port on existing Standard clusterssection.

Check the port status in Standard node pools

Run the following command:

gcloudcontainernode-poolsdescribeNODE_POOL_NAME\--cluster=CLUSTER_NAME\--location=LOCATION\--flatten=config\--format="value(kubeletConfig)"

Replace the following:

• NODE_POOL_NAME: the name of your node pool.
• CLUSTER_NAME: the name of the cluster.
• LOCATION: the location of the cluster.

If the port is disabled, the output is the following:

insecureKubeletReadonlyPortEnabled: false

Prevent insecure port usage with an organization policy

You can use Organization Policy Service to enforce that clusters in your organization,folder, or project must disable the insecurekubelet read-only port. Toenforce this requirement, you create acustom constraint,and then reference that constraint in an organization policy. To deny creationor update operations when the insecurekubelet read-only port is enabled,see these examples:

• The following custom constraint denies cluster creation or update operationsif the insecurekubelet read-only port is enabled:

  name:organizations/ORGANIZATION_ID/customConstraints/custom.disableClusterInsecureKubeletReadOnlyPortresourceTypes:-container.googleapis.com/ClustermethodTypes:-CREATE-UPDATEcondition:"resource.NodeKubeletConfig.insecureKubeletReadonlyPortEnabled==true||resource.AutoprovisioningNodePoolDefaults.insecureKubeletReadonlyPortEnabled==true||resource.NodePoolAutoConfig.NodeKubeletConfig.insecureKubeletReadonlyPortEnabled==true"actionType:DENYdisplayName:Disable insecure kubelet read-only portdescription:All new and existing clusters must disable the insecure kubelet read-only port.

  ReplaceORGANIZATION_ID with your organization ID.

• The following custom constraint denies node pool creation or update operationsif the insecurekubelet read-only port is enabled:

  name:organizations/ORGANIZATION_ID/customConstraints/custom.disableNodePoolInsecureKubeletReadOnlyPortresourceTypes:-container.googleapis.com/NodePoolmethodTypes:-CREATE-UPDATEcondition:"resource.NodeConfig.NodeKubeletConfig.insecureKubeletReadonlyPortEnabled==true"actionType:DENYdisplayName:Disable insecure kubelet read-only portdescription:All new and existing node pools must disable the insecure kubelet read-only port.

  ReplaceORGANIZATION_ID with your organization ID.

For more information about how to create a custom constraint and enforce theconstraint in an organization policy, seeRestrict actions on GKE resources using custom organization policies.