You can control whether your Google Kubernetes Engine (GKE) nodes are physicallylocated relative to each other within azoneby using acompact placement policy.

Overview

When you create node pools and workloads in a GKE cluster, you can define acompact placement policy, which specifies that these nodes orworkloads should be placed in closer physical proximity to each other within azone. Having nodes closer toeach other can reduce network latency between nodes, which can be especiallyuseful for tightly-coupled batch workloads.

Use compact placement with GKE Autopilot

In Autopilot clusters, you can request compact placement for specificworkloads by adding node selectors to your Pod specification. You can use thedefault Autopilot compact placement policy or an existingCompute Engine compact placement policy that uses the N2 machineseries or the N2D machine series.

Limitations

• GKE provisions workloads within a compact placement in thesame zone.
• Available with theBalanced,Performance andAccelerator computeclasses.
• Available only on C2, C2D, C3, C3D, C4D (1.33.0-gke.1439000 or later), H3,H4D, N2, and N2Dmachine types.
• Available only withA100, L4, and H100GPUs.
• Compact placement is available for Pods grouped on up to 1,500 nodes.

Enable a compact placement policy

To enable compact placement for GKE Autopilot, add anodeSelector to the Pod specification with the following keys:

• cloud.google.com/gke-placement-group: the identifier you assign for the group of Podsthat should run together, in the same compact placement group. Each placementgroup is limited to 1,500 nodes. We recommend you limit a placement group onlyto the workloads that benefit from the grouping, and distribute yourworkloads into separate placement groups where possible.

• One of the following keys to define the type of resource:

  • cloud.google.com/compute-class: "Balanced"
  • cloud.google.com/gke-accelerator: "nvidia-tesla-a100"

• cloud.google.com/placement-policy-name: optionally, the name of anexisting Compute Engine compact placement policy. You can onlyspecify a custom compact placement policy in GKE version1.31.1-gke.2010000 and later.

  For instructions, see theCreate a compact placement policy section on this page.

The following example Pod specification enables compact placement with a customcompact placement policy:

apiVersion:v1kind:Podmetadata:# lines omitted for clarityspec:nodeSelector:cloud.google.com/gke-placement-group:"placement-group-1"cloud.google.com/compute-class:"Balanced"cloud.google.com/placement-policy-name:PLACEMENT_POLICY_NAME

ReplacePLACEMENT_POLICY_NAME with the name of anexisting Compute Engine compact placement policy. To use thedefault compact placement policy for Autopilot, omit thecloud.google.com/placement-policy-name line.

Use a custom compact placement policy without placement groups

To use a custom compact placement policy without placement groups, you musteither add thecloud.google.com/placement-policy-name node selector to your Podspecification, or define acustom compute class with placement priority.

Add thecloud.google.com/placement-policy-name node selector to your Pod specification

This approach might be useful if you want to use aJobSetto schedule each job separately, but you also want to use a custom compactplacement policy to place the nodes that run the same job closer to each other.

Because JobSet doesn't support specifying different node selectors for each job,you can't use JobSet with placement groups in this scenario. However, you canuse JobSet's built-in support for exclusive topologies to achieve the sameeffect.

The following example Pod specification enables compact placement with a customcompact placement policy for a JobSet workload:

apiVersion:jobset.x-k8s.io/v1alpha2kind:JobSetmetadata:name:my-jobsetannotations:alpha.jobset.sigs.k8s.io/exclusive-topology:cloud.google.com/gke-nodepoolspec:replicatedJobs:-name:my-jobtemplate:spec:# lines omitted for claritytemplate:spec:nodeSelector:cloud.google.com/placement-policy-name:PLACEMENT_POLICY_NAMEcloud.google.com/machine-family:"n2"# lines omitted for clarity

ReplacePLACEMENT_POLICY_NAME with the name of anexisting Compute Engine compact placement policy.

Use a custom compute class to define placement policy

Refer tocustom compute class with placement prioritydocumentation to see the specifics.

Instead of adding anodeSelector with policy name directly to your workloadspecification, you can apply a compact placement policy by setting theplacementfield in aComputeClass definition. For details, seecustom compute class with placement priority.

Use compact placement with GKE Standard

Limitations

Compact placement in GKE Standard node pools has the following limitations:

• Supported only in new node pools. You cannot enable or disable compactplacement on existing node pools.
• Available only fornode pools operating in a singlezone.
• Available only on A2, A3, A4, C2, C2D, C3, C3D, C4, C4D, G2, G4, H3, H4D,N2, and N2Dmachine types.However, forA3Ultra andA4, we recommendusing block-targeted reservations instead of compact placement. For details,seeReservecapacity.
• Supports up to 1,500Compute Engine VM instancesin each policy. Any node pool that exceeds this limit at any time will berejected during creation.
• Supplying a custom resource policy using theplacement-policy flag is notsupported with blue-green upgrades.

Create a compact placement policy

To create compact placement policies, in the Google Cloud CLI, you specify theplacement-type=COMPACT option during node pool or cluster creation. With thissetting, GKE attempts to place nodes within a node pool in closerphysical proximity to each other.

To use an existingresource policy in yourcluster, specify the location of your custom policy for theplacement-policy flag during node pool or clustercreation. This enables the flexibility of using reserved placements, multiple node pools with the same placement policy, and other advanced placement options. However, it also requires more manual operations than specifying the --placement-type=COMPACT flag. For example, you need to create, delete, and maintain your custom resource policies. Make sure thatthe maximum number of VM instances is respected acrossall node pools using the resource policy. If this limit is reached while some ofyour node pools haven't reach their maximum size, adding any more nodes will fail.

If you don't specify theplacement-type andplacement-policy flags, then bydefault there are no requirements on node placement.

Create a compact placement policy in a new cluster

When you create a new cluster, you can specify a compact placement policy thatwill be applied to the default node pool. Any subsequent node pools that youcreate for the cluster, you will need to specify whether to apply compactplacement.

To create a new cluster where the default node pool has a compact placementpolicy applied, use the following command:

gcloudcontainerclusterscreateCLUSTER_NAME\--machine-typeMACHINE_TYPE\--placement-typeCOMPACT\--max-surge-upgrade0\--max-unavailable-upgradeMAX_UNAVAILABLE

Replace the following:

• CLUSTER_NAME: The name of your new cluster.
• MACHINE_TYPE: The type of machine to use for nodes,which must be a supported machine type listed in the Standardclusterlimitations.
• --placement-type COMPACT: Applies compact placement for thenodes in the default node pool.
• MAX_UNAVAILABLE: Maximum number of nodes that can be unavailable at the same time during a node pool upgrade. For compact placement we recommendfast no surge upgrades to optimize the likelihood of finding colocated nodes during upgrades.

Create a compact placement policy on an existing cluster

On an existing cluster, you can create a node pool that has a compact placementpolicy applied.

To create a node pool that has a compact placement policy applied, use thefollowing command:

gcloudcontainernode-poolscreateNODEPOOL_NAME\--machine-typeMACHINE_TYPE\--clusterCLUSTER_NAME\--placement-typeCOMPACT\--max-surge-upgrade0\--max-unavailable-upgradeMAX_UNAVAILABLE

Replace the following:

• NODEPOOL_NAME: The name of your new node pool.
• MACHINE_TYPE: The type of machine to use for nodes,which must be a supported machine type listed in the Standardclusterlimitations.
• CLUSTER_NAME: The name of your existing cluster.
• --placement-type COMPACT: Indicates to apply compact placementfor the nodes in the new node pool.
• MAX_UNAVAILABLE: Maximum number of nodes that can be unavailable at the same time during a node pool upgrade. For compact placement we recommendfast no surge upgrades to optimize the likelihood of finding colocated nodes during upgrades.

Create node pools using a shared custom placement policy

You can manually create aresource policy and use it in multiple node pools.

1. Create the resource policy in the cluster Google Cloud region:

   gcloudcomputeresource-policiescreategroup-placementPOLICY_NAME\--regionREGION\--collocationcollocated

   Replace the following:

   • POLICY_NAME: The name of your resource policy.
   • REGION: The region of your cluster.

2. Create a node pool using the custom resource policy:

   gcloudcontainernode-poolscreateNODEPOOL_NAME\--machine-typeMACHINE_TYPE\--clusterCLUSTER_NAME\--placement-policyPOLICY_NAME\--max-surge-upgrade0\--max-unavailable-upgradeMAX_UNAVAILABLE

   Replace the following:

   • NODEPOOL_NAME: The name of your new node pool.
   • MACHINE_TYPE: The type of machine to use fornodes, which must be a supported machine type listed in theStandard clusterlimitations.
   • CLUSTER_NAME: The name of your existing cluster.
   • MAX_UNAVAILABLE: Maximum number of nodes that can be unavailable at the same time during a node pool upgrade. For compact placement we recommendfast no surge upgrades to optimize the likelihood of finding colocated nodes during upgrades.

Use a Compute Engine reservation with a compact placement policy

Reservations help you guarantee that hardware is available in a specified zone, reducing the risk of node pool creation failure caused by insufficient hardware.

1. Create a reservation that specifies a compact placement policy:

   gcloudcomputereservationscreateRESERVATION_NAME\--vm-countMACHINE_COUNT\--machine-typeMACHINE_TYPE\--resource-policiespolicy=POLICY_NAME\--zoneZONE\--require-specific-reservation

   Replace the following:

   • RESERVATION_NAME: The name of your reservation.
   • MACHINE_COUNT: The number of reserved nodes.
   • MACHINE_TYPE: The type of machine to use fornodes, which must be a supported machine type listed in theStandard clusterlimitations.
   • POLICY_NAME: The name of your resource policy.
   • ZONE: The zone where to create your reservation.

2. Create a node pool by specifying both the compact placement policy and thereservation you created in the previous step:

   gcloudcontainernode-poolscreateNODEPOOL_NAME\--machine-typeMACHINE_TYPE\--clusterCLUSTER_NAME\--placement-policyPOLICY_NAME\--reservation-affinityspecific\--reservationRESERVATION_NAME\--max-surge-upgrade0\--max-unavailable-upgradeMAX_UNAVAILABLE

Replace the following:

• NODEPOOL_NAME: The name of your new node pool.
• MACHINE_TYPE: The type of machine to use for nodes,which must be a supported machine type listed in the Standardclusterlimitations.
• CLUSTER_NAME: The name of your existing cluster.

Create a workload on nodes that use compact placement

To run workloads on dedicated nodes that use compact placement, you can useseveral Kubernetes mechanisms, such asassigning pods to nodes andpreventing scheduling unwanted pods on a group of nodes to achieve this.

In the following example, we add ataintto the dedicated nodes and add a corresponding toleration and affinity to the Pods.

1. Add a taint to nodes in the node pool that has a compact placement policy:

   kubectltaintnodes-lcloud.google.com/gke-nodepool=NODEPOOL_NAMEdedicated-pool=NODEPOOL_NAME:NoSchedule

2. In the workload definition, specify the necessary toleration and a nodeaffinity. Here's an example with a single Pod:

   apiVersion:v1kind:Podmetadata:...spec:...tolerations:-key:dedicated-pooloperator:"Equal"value:"NODEPOOL_NAME"effect:"NoSchedule"affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:-matchExpressions:-key:dedicated-pooloperator:Invalues:-NODEPOOL_NAME

In some locations, it might not be possible to create a large node pool using acompact placement policy. To limit the size of such node pools to what'snecessary, you should consider creating a node pool per workload requiringcompact placement.

Use compact placement for node auto-provisioning

With node auto-provisioning, GKE automatically provisionsnode pools based on cluster resource demand. For more information, seeUsing node auto-provisioning.

To enable compact placement for node auto-provisioning, add anodeSelector tothe Pod specification like in the following example:

apiVersion:v1kind:Podmetadata:# lines omitted for clarityspec:nodeSelector:cloud.google.com/gke-placement-group:PLACEMENT_GROUP_IDENTIFIERcloud.google.com/machine-family:MACHINE_FAMILYcloud.google.com/placement-policy-name:PLACEMENT_POLICY_NAME# lines omitted for clarity

Replace the following:

• PLACEMENT_GROUP_IDENTIFIER: the identifier that youassign for the group of Pods that should run together, in the same compactplacement group.
• MACHINE_FAMILY: the name of the machine family. Useone ofthe machine families that support compact placement.We recommend you use C2 or C2D machine families for workloads with compute andnetworking performance requirements.

• PLACEMENT_POLICY_NAME: optionally, the name of anexisting Compute Engine compact placement policy. GKEuses the specified compact placement policy when node auto-provisioningcreates a new node pool to group your Pods together. You can only specify acustom compact placement policy in GKE version1.31.1-gke.2010000 and later.

  For instructions, see theCreate a compact placement policy section on this page.

You can omit thecloud.google.com/machine-family key if the Pod configuration alreadydefines a machine type supported with compact placement. For example, if thePod specification includesnvidia.com/gpu and the cluster is configured touse A100 GPUs, you don't need to include thecloud.google.com/machine-family key.

The following example is a Pod specification that definesnvidia.com/gpu request and the cluster is configured to use A100 GPUs. ThisPodspec doesn't include thecloud.google.com/machine-family key:

apiVersion:v1kind:Podmetadata:...spec:...nodeSelector:cloud.google.com/gke-placement-group:PLACEMENT_GROUP_IDENTIFIERcloud.google.com/gke-accelerator:"nvidia-tesla-a100"resources:limits:nvidia.com/gpu:2

To learn more, seehow to configure Pods to consume GPUs.

Optimize placement group size

Because GKE finds the best placement for smaller deployments, werecommend you instruct GKE to avoid running different type of Pods in the sameplacement group. Add a toleration key with thecloud.google.com/gke-placement-group key and the compact placement identifieryou defined.

The following example is a Pod specification that defines a Pod toleration withcompact placement:

apiVersion:v1kind:Podmetadata:...spec:...tolerations:-key:cloud.google.com/gke-placement-groupoperator:"Equal"value:PLACEMENT_GROUP_IDENTIFIEReffect:"NoSchedule"

For more information about node auto-provisioning with Pod toleration, seeWorkload separation

What's next

• Learn aboutDefining instance placement policiesin Compute Engine.