This document shows you how to customize your Google Kubernetes Engine (GKE)node configuration by using a configuration file called anode system configuration.

A node system configuration is a configuration file that provides a way toadjust a limited set of system settings. In your node pool, you can use a nodesystem configuration to specify custom settings for thekubelet Kubernetes node agent and forsysctl low-level Linux kernelconfigurations.

• Runtime configuration file: to customize a containerd container runtimeon your GKE nodes, you can use a different file called aruntime configuration file. For more information, seeCustomize containerd configuration in GKE nodes.
• ComputeClass: you can specify node attributes in your GKEComputeClass specification. You can use ComputeClasses, in bothGKE Autopilot mode and Standard mode, inGKE version 1.32.1-gke.1729000 and later.For more information, seeCustomize the node system configuration.
• DaemonSets: you can also use DaemonSets to customize nodes. For moreinformation, seeAutomatically bootstrapping GKE nodes with DaemonSets.

Node system configurations arenot supported in Windows Server nodes.

Before you begin

Before you begin, make sure to do the following:

• Install command-line tools:
  • If you use the gcloud CLI examples in this document, ensure thatyou install and configure theGoogle Cloud CLI.
  • If you use the Terraform examples, ensure that you install and configureTerraform.
• Grant permissions: you need appropriate IAM permissions tocreate and update GKE clusters and node pools, such ascontainer.clusterAdmin or a different role with equivalent permissions.
• Plan for potential workload disruption: custom node configurations areapplied at the node pool level. Changes typically trigger a rolling updateof the nodes in the pool, which involvesre-creating the nodes.Plan for potential workload disruption and use Pod Disruption Budgets (PDBs)where appropriate.
• Back up and test all changes: always test configuration changes in astaging or development environment before you apply them to production.Incorrect settings can lead to node instability or workload failures.
• Review GKE default settings: GKE nodeimages come with optimized default configurations. Only customize parametersif you have a specific need and understand the impact of your changes.

Use a node system configuration in GKE Standard mode

When you use a node system configuration, you use a YAML file that contains theconfiguration parameters for thekubelet and the Linux kernel. Although nodesystem configurations are also available in GKEAutopilot mode, the steps in this document show you how to create anduse a configuration file for GKE Standard mode.

To use a node system configuration in GKE Standardmode, do the following:

1. Create a configuration file. This file contains yourkubeletandsysctl configurations.
2. Add the configuration when you create a cluster, or when you createor update a node pool.

Create a configuration file

Write your node system configuration in YAML. The following example addsconfigurations for thekubelet andsysctl options:

kubeletConfig:cpuManagerPolicy:staticallowedUnsafeSysctls:-'kernel.shm*'-'kernel.msg*'-'kernel.sem'-'fs.mqueue*'-'net.*'linuxConfig:sysctl:net.core.somaxconn:'2048'net.ipv4.tcp_rmem:'4096873806291456'

In this example, the following applies:

• ThecpuManagerPolicy: static field configures thekubelet to use thestatic CPU managementpolicy. + Thenet.core.somaxconn: '2048' field limits thesocket listen() backlog to2,048 bytes.
• Thenet.ipv4.tcp_rmem: '4096 87380 6291456' field sets the minimum,default, and maximum value of the TCP socket receive buffer to 4,096 bytes,87,380 bytes, and 6,291,456 bytes, respectively.

If you want to add configurations only for thekubelet orsysctl, includeonly that section in your node system configuration. For example, to add akubelet configuration, create the following file:

kubeletConfig:cpuManagerPolicy:static

For a complete list of the fields that you can add to your node systemconfiguration, see theKubelet configuration options andSysctl configuration options sections.

Add the configuration to a Standard node pool

After you create the node system configuration, add the--system-config-from-fileflag by using the Google Cloud CLI. You can add this flag when you create acluster, or when you create or update a node pool. You can't add a node systemconfiguration by using the Google Cloud console.

Create a cluster with the node system configuration

You can add a node system configuration during cluster creation by using thegcloud CLI or Terraform. The following instructions apply the nodesystem configuration to the default node pool:

gcloudcontainerclusterscreateCLUSTER_NAME\--location=LOCATION\--system-config-from-file=SYSTEM_CONFIG_PATH

resource "google_container_cluster" "default" {  name     = "gke-standard-regional-cluster"  location = "us-central1"  initial_node_count = 1  node_config {    # Kubelet configuration    kubelet_config {      cpu_manager_policy = "static"    }    linux_node_config {      # Sysctl configuration      sysctls = {        "net.core.netdev_max_backlog" = "10000"      }      # Linux cgroup mode configuration      cgroup_mode = "CGROUP_MODE_V2"      # Linux huge page configuration      hugepages_config {        hugepage_size_2m = "1024"      }    }  }}

Create a new node pool with the node system configuration

You can add a node system configuration when you use the gcloud CLIor Terraform to create a new node pool.

The following instructions apply the node system configuration to a new nodepool:

gcloudcontainernode-poolscreatePOOL_NAME\--clusterCLUSTER_NAME\--location=LOCATION\--system-config-from-file=SYSTEM_CONFIG_PATH

resource "google_container_node_pool" "default" {  name    = "gke-standard-regional-node-pool"  cluster = google_container_cluster.default.name  node_config {    # Kubelet configuration    kubelet_config {      cpu_manager_policy = "static"    }    linux_node_config {      # Sysctl configuration      sysctls = {        "net.core.netdev_max_backlog" = "10000"      }      # Linux cgroup mode configuration      cgroup_mode = "CGROUP_MODE_V2"      # Linux huge page configuration      hugepages_config {        hugepage_size_2m = "1024"      }    }  }}

Update the node system configuration of an existing node pool

You can update the node system configuration of an existing node pool by runningthe following command:

gcloudcontainernode-poolsupdatePOOL_NAME\--cluster=CLUSTER_NAME\--location=LOCATION\--system-config-from-file=SYSTEM_CONFIG_PATH

Replace the following:

• POOL_NAME: the name of the node pool that you want toupdate
• CLUSTER_NAME: the name of the cluster that you want toupdate
• LOCATION: the compute zone or region of the cluster
• SYSTEM_CONFIG_PATH: the path to the file that containsyourkubelet andsysctl configurations

This change requires re-creating the nodes, which can cause disruption to yourrunning workloads. For more information about this specific change, find thecorresponding row in themanual changes that re-create the nodes using a node upgrade strategy without respecting maintenance policiestable.

For more information about node updates, seePlanning for node update disruptions.

Edit a node system configuration

To edit a node system configuration, you can create a new node pool with theconfiguration that you want, or update the node system configuration of anexisting node pool.

Edit by creating a node pool

To edit a node system configuration by creating a node pool, do the following:

1. Create a configuration file with the configuration that you want.
2. Add the configuration to a new node pool.
3. Migrate your workloads to the new node pool.
4. Delete the old nodepool.

Edit by updating an existing node pool

To edit the node system configuration of an existing node pool, follow theinstructions in theUpdate node pool tab foradding the configuration to a node pool. When you update a node systemconfiguration and the new configuration overrides the node pool's existingsystem configuration, the nodes must be re-created. If you omit any parametersduring an update, the parameters are set to their respective defaults.

If you want to reset the node system configuration back to the defaults, updateyour configuration file with empty values for thekubelet andsysctl fields,for example:

kubeletConfig:{}linuxConfig:sysctl:{}

Delete a node system configuration

To remove a node system configuration, do the following steps:

1. Create a node pool.
2. Migrate your workloads to the new node pool.
3. Delete the node poolthat has the old node system configuration.

Configuration options for thekubelet

The tables in this section describe thekubelet options that you can modify.

CPU management

The following table describes the CPU management options for thekubelet.

Memory management and eviction

The following table describes the modifiable options for memory management andeviction. This section also contains a separate table that describes themodifiable options for theevictionSoft flag.

The following table shows the modifiable options for theevictionSoft flag.The same options also apply to theevictionSoftGracePeriod andevictionMinimumReclaim flags with different restrictions.

PID management

The following table describes the modifiable options for PID management.

Logging

The following table describes the modifiable options for logging.

Image garbage collection

The following table describes the modifiable options for image garbagecollection.

Image pulling

The following table describes the modifiable options for image pulling.

Security and unsafe operations

The following table describes the modifiable options for configuring securityand handling unsafe operations.

Resource Managers

Kubernetes offers a suite of Resource Managers. You can configure these ResourceManagers to coordinate and optimize the alignment of node resources for Podsthat are configured with specific requirements for CPUs, devices, and memory(hugepages) resources.

The following table describes the modifiable options for Resource Managers.

Sysctl configuration options

To tune the performance of your system, you can modify Linux kernel parameters.The tables in this section describe the various kernel parameters that you canconfigure.

Filesystem parameters (fs.*)

The following table describes the modifiable parameters for the Linuxfilesystem. These settings control the behavior of the Linux filesystem, such asfile handle limits and event monitoring.

Kernel parameters (kernel.*)

The following table describes the modifiable parameters for the Linux kernel.These settings configure core kernel functionalities, including shared memoryallocation.

Network parameters (net.*)

The following table describes the modifiable parameters for networking. Thesesettings tune the performance and behavior of the networking stack, from socketbuffers to connection tracking.

Virtual Memory parameters (vm.*)

The following table describes the modifiable parameters for the Virtual Memorysubsystem. These settings manage the Virtual Memory subsystem, which controlshow the kernel handles memory, swapping, and disk caching.

For more information about the supported values for eachsysctl flag, see the--system-config-from-file gcloud CLI documentation.

DifferentLinux namespacesmight have unique values for a givensysctl flag, but others might be globalfor the entire node. Updatingsysctl options by using a node systemconfiguration helps ensure that thesysctl is applied globally on the node andin each namespace, so that each Pod has identicalsysctl values in each Linuxnamespace.

Linux cgroup mode configuration options

The container runtime andkubelet use Linux kernelcgroups for resource management, suchas limiting how much CPU or memory each container in a Pod can access. There aretwo versions of the cgroup subsystem in the kernel:cgroupv1 andcgroupv2.Kubernetes support forcgroupv2 was introduced as alpha in Kubernetes version1.18, beta in 1.22, and GA in 1.25. For more information, see theKubernetes cgroups v2 documentation.

Node system configuration lets you customize the cgroup configuration of yournode pools. You can usecgroupv1 orcgroupv2. GKE usescgroupv2 for new Standard node pools that run version 1.26 and later,andcgroupv1 for node pools that run versions earlier than 1.26. For nodepools that were created with node auto-provisioning, the cgroup configurationdepends on the initial cluster version, not the node pool version.cgroupv1 isnot supported on Arm machines.

You can use node system configuration to change the setting for a node pool tousecgroupv1 orcgroupv2 explicitly. Upgrading an existing node pool thatusescgroupv1 to version 1.26 doesn't change the setting tocgroupv2.Existing node pools that run a version earlier than 1.26—and that don'tinclude a customized cgroup configuration—will continue to usecgroupv1.To change the setting, you must explicitly specifycgroupv2 for the existingnode pool.

For example, to configure your node pool to usecgroupv2, use a node systemconfiguration file such as the following:

linuxConfig:cgroupMode:'CGROUP_MODE_V2'

The supportedcgroupMode options are the following:

• CGROUP_MODE_V1: usecgroupv1 on the node pool.
• CGROUP_MODE_V2: usecgroupv2 on the node pool.
• CGROUP_MODE_UNSPECIFIED: use the default GKE cgroupconfiguration.

To usecgroupv2, the following requirements and limitations apply:

• For a node pool that runs a version earlier than 1.26, you must usegcloud CLI version408.0.0 or later. Alternatively,usegcloud beta with version395.0.0 or later.
• Your cluster and node pools must run GKE version1.24.2-gke.300 or later.
• You must use either the Container-Optimized OS with containerd orUbuntu with containerdnode image.
• If any of your workloads depend on reading the cgroup filesystem(/sys/fs/cgroup/...), ensure that they are compatible with thecgroupv2API.
• If you use any monitoring or third-party tools, ensure that they arecompatible withcgroupv2.
• If you use Java workloads (JDK), we recommend that you use versions whichfully support cgroupv2,including JDK8u372, JDK 11.0.16 or later, or JDK 15 or later.

Verify cgroup configuration

When you add a node system configuration, GKE must re-create thenodes to implement the changes. After youadd the configuration to a node pool and the nodes are re-created, youcan verify the new configuration.

You can verify the cgroup configuration for nodes in a node pool by usinggcloud CLI or thekubectl command-line tool:

gcloudcontainernode-poolsdescribePOOL_NAME\--format='value(Config.effectiveCgroupMode)'

Linux hugepages configuration options

You can use a node system configuration file to pre-allocatehugepages. Kubernetessupports pre-allocated hugepages as a resource type, similar to CPU or memory.

To use hugepages, the following limitations and requirements apply:

• To ensure that the node is not fully occupied by hugepages, the overall sizeof the allocated hugepages can't exceed either the following:
  • On machines with less than 30 GB memory: 60% of the totalmemory. For example, on e2-standard-2 machine with 8 GB of memory,you can't allocate more than 4.8 GB for hugepages.
  • On machines with more than 30 GB memory: 80% of the totalmemory. For example, on c4a-standard-8 machines with 32 GB ofmemory, hugepages cannot exceed 25.6 GB.
• 1 GB hugepages are only available on A3, C2D, C3, C3D, C4, C4A, C4D,CT5E, CT5LP, CT6E, H3, M2, M3, M4, or Z3 machine types.

The following table describes the modifiable settings for Linux hugepages.

Transparent hugepages (THP)

You can use a node system configuration file to enable the Linux kernel'sTransparent HugePage Support.With THP, the kernel automatically assigns hugepages to processes withoutmanual pre-allocation.

The following table describes the modifiable parameters for THP.

THP is available on GKE version 1.33.2-gke.4655000 or later. Itis also enabled on new TPU node pools by default on GKE version1.33.2-gke.4655000 or later. THP isn't enabled when you upgrade existing nodepools to a supported version or later.

What's next

• Learn more about node pools.
• Learn how to create node pools.