This document shows you how to prepare your Google Kubernetes Engine (GKE)environment to enable the multi-clusterGKE Gatewaycontroller. This controller is aGoogle-hosted controller that provisions external and internal load balancersfor your GKE clusters. To learn how to use Gateway resources for containerload balancing, seeDeployingGateways orDeployingmulti-clusterGateways.

The multi-cluster GKE Gateway Controller installs the followingmulti-cluster GatewayClasses in your clusters:

• gke-l7-global-external-managed-mc for global external multi-clusterGateways
• gke-l7-regional-external-managed-mc for regional external multi-cluster
• gke-l7-cross-regional-internal-managed-mc for cross-regional internal multi-clusterGateways
• gke-l7-rilb-mc for regional internal multi-cluster Gateways
• gke-l7-gxlb-mc for global external Classic multi-cluster Gateways

Learn more about the capabilities of the variousGatewayClasses in GKE.

Pricing

All Compute Engine resources deployed through the Gateway controllers are chargedagainst the project in which your GKE clusters reside. Thesingle-cluster Gateway controller is offered at no additional charge as a part ofGKE Standard and Autopilot pricing. Pricing formulti-cluster Gateways is described in theMulti Cluster Gateway and Multi Cluster Ingress pricing page.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

GKE Gateway controller requirements

• Gateway API is supported onVPC-nativeclusters only.
• If you are using the regional or cross-region GatewayClasses, you must enable aproxy-only subnet.
• Your cluster must have theHttpLoadBalancing add-on enabled.
• If you are using Istio, you must upgrade Istio to one of the followingversions:
  • 1.15.2 or later
  • 1.14.5 or later
  • 1.13.9 or later.
• If you are using Shared VPC, then in the host project, you need to assign theCompute Network User role to the GKE Service account for the service project.

Multi-cluster Gateway requirements

In addition to the GKE Gateway controller requirements, formulti-cluster Gateway deployments, make sure you have performed the following tasks:

• Enable the GatewayAPIon your config cluster.
• EnableWorkload Identity Federation for GKEon your clusters.
• Complete thefleet general prerequisitesfor registering your clusters.

• Enable the following multi-cluster Gateways required APIs in your project:

  • Cloud Service Mesh API
  • Multi-cluster Services API
  • Multi Cluster Ingress API

  Run this command to enable the required APIs if they are not already enabled:

  gcloudservicesenable\trafficdirector.googleapis.com\multiclusterservicediscovery.googleapis.com\multiclusteringress.googleapis.com\--project=PROJECT_ID

  ReplacePROJECT_ID with the project ID where yourGKE clusters are running.

Restrictions and limitations

The samelimitations and known issuesfor single-cluster Gateways also apply to multi-cluster Gateways.

In addition to the single-cluster Gateway limitations, the following limitationsapply to multi-cluster Gateways:

• Load balancing to backends in different regions is not supported with theregional internal GatewayClassgke-l7-rilb-mc. To learn more about thedifferent features supported with each GatewayClass,seeGatewayClass capabilities.

• AService is not supported as abackendRefs with multi-cluster Gateway.Multi-cluster Gateway only supportsServiceImport as a validbackendRefs.

• All clusters in the fleet must exist in thefleet host project.

• Cross-project load balancing is not supported. All clusters (config cluster andtarget clusters) attached to the same multi-cluster Gateway must be deployedin the same Shared VPC host project or service project. For moreinformation on supported Shared VPC topologies with multi-clusterGateway, seeUse multi-cluster Gateway with Shared VPC.

• Cross-VPC load balancing is not supported. All clusters (config cluster andtarget clusters) attached to the same multi-cluster Gateway must be deployedin the same VPC.

• Multi-cluster Gateway depends on MCS for handling cross-cluster servicediscovery. As such, services multi-cluster Gateway exposes are subject toall theMulti-cluster Servicerequirements.

• Multi-cluster Gateways can leak load balancer resources under the followingscenarios:

  • The Fleet ingress feature is updated with a new config clusterwhich does not have allGateway resources which exist on the currentconfig cluster.
  • The Fleet ingress feature is disabled whileGateway resourceswhich reference a multi-clusterGatewayClass exist on the configcluster.

• Multi-cluster Gateway runs as a global service. If the multi-cluster Gatewaycontroller encounters a regional Fleet (Hub) control plane failure, itresponds by failing statically and making no further load balancer changesuntil the region returns to service.

Quotas

GKE Gateway usesCloud Load Balancing quotas tolimit the number of resources that the Gateway controller can create to manageingress traffic routed to GKE clusters.

Set up your environment for multi-cluster Gateways

It requires multiple GKE clusters to complete the examples inDeploying multi-cluster Gateways.All of the clusters are registered to the samefleet so that multi-clusterGateways and Services can operate across them.

The following steps will deploythree GKE clusters across two different regions in your project:

• us-west1-a/gke-west-1
• us-west1-a/gke-west-2
• us-east1-b/gke-east-1

This will create the following cluster topology:

These GKE clusters are used to demonstratemulti-region load balancing andblue-green, multi-cluster traffic splittingusing external and internal Gateways.

Deploy clusters

In these steps you will deploy three GKE clusters into regionsus-east1 andus-west1.

Clusters will be registered to your project'sfleet.Grouping your GKE clusters together in a fleet allows them to be targeted by a multi-cluster Gateway.

1. Create a GKE cluster inus-west1namedgke-west-1:

   gcloudcontainerclusterscreategke-west-1\--gateway-api=standard\--location=us-west1-a\--workload-pool=PROJECT_ID.svc.id.goog\--cluster-version=VERSION\--enable-fleet\--project=PROJECT_ID

   Replace the following:

   • PROJECT_ID: the project ID where yourGKE clusters are running.
   • VERSION: the GKE version, 1.24 or later.

2. Create another GKE cluster inus-west1 (or the same region as the previous cluster)namedgke-west-2:

   gcloudcontainerclusterscreategke-west-2\--gateway-api=standard\--location=us-west1-a\--workload-pool=PROJECT_ID.svc.id.goog\--cluster-version=VERSION\--enable-fleet\--project=PROJECT_ID

3. Create a GKE cluster inus-east1 (or a region that isdifferent than the previous one) namedgke-east-1

   gcloudcontainerclusterscreategke-east-1\--gateway-api=standard\--location=us-east1-b\--workload-pool=PROJECT_ID.svc.id.goog\--cluster-version=VERSION\--enable-fleet\--project=PROJECT_ID

4. Confirm that the clusters have been successfully registered to the fleet:

   gcloudcontainerfleetmembershipslist--project=PROJECT_ID

   The output will be similar to the following:

   NAME        EXTERNAL_ID                           LOCATIONgke-east-1  45a80b37-4b00-49aa-a68b-b430fce1e3f0  us-east1gke-west-2  ac7087a5-f5ee-401e-b430-57f3af141239  us-west1gke-west-1  549efe3a-b18e-4eb9-8796-e50b7967cde2  us-west1

Configure cluster credentials

This step configures cluster credentials with memorable names. This makes iteasier to switch between clusters when deploying resources across severalclusters.

1. Fetch the credentials for clustergke-west-1,gke-west-2, andgke-east-1:

   gcloudcontainerclustersget-credentialsgke-west-1--location=us-west1-a--project=PROJECT_IDgcloudcontainerclustersget-credentialsgke-west-2--location=us-west1-a--project=PROJECT_IDgcloudcontainerclustersget-credentialsgke-east-1--location=us-east1-b--project=PROJECT_ID

   This stores the credentials locally so that you can use your kubectl clientto access the cluster API servers. By default an auto-generated name iscreated for the credential.

2. Rename the cluster contexts so they are easier to reference later:

   kubectlconfigrename-contextgke_PROJECT_ID_us-west1-a_gke-west-1gke-west-1kubectlconfigrename-contextgke_PROJECT_ID_us-west1-a_gke-west-2gke-west-2kubectlconfigrename-contextgke_PROJECT_ID_us-east1-b_gke-east-1gke-east-1

   ReplacePROJECT_ID with the project ID where yourclusters are deployed.

Enable multi-cluster Services in the fleet

1. Enable multi-cluster Services in your fleet for the registered clusters.This enables the MCS controller for the three clusters that are registered toyour fleet so that it can start listening to and exporting Services.

   gcloudcontainerfleetmulti-cluster-servicesenable\--projectPROJECT_ID

2. Grant Identity and Access Management (IAM) permissions required by the MCS controller:

   gcloudprojectsadd-iam-policy-bindingPROJECT_ID\--member"serviceAccount:PROJECT_ID.svc.id.goog[gke-mcs/gke-mcs-importer]"\--role"roles/compute.networkViewer"\--project=PROJECT_ID

   ReplacePROJECT_ID with the project ID where yourclusters are deployed.

3. Confirm that MCS is enabled for the registered clusters. You will see thememberships for the three registered clusters. It may take several minutesfor all of the clusters to show.

   gcloudcontainerfleetmulti-cluster-servicesdescribe--project=PROJECT_ID

   The output is similar to the following:

   createTime: '2023-10-12T06:14:33.466903587Z'membershipStates:  projects/441323991697/locations/us-east1/memberships/gke-east-1:    state:      code: OK      description: Firewall successfully updated      updateTime: '2023-10-12T06:15:28.395318091Z'  projects/441323991697/locations/us-west1/memberships/gke-west-1:    state:      code: OK      description: Firewall successfully updated      updateTime: '2023-10-12T06:15:30.534594027Z'  projects/441323991697/locations/us-west1/memberships/gke-west-2:    state:      code: OK      description: Firewall successfully updated      updateTime: '2023-10-12T06:15:29.110582109Z'name: projects/pierre-louis-playground/locations/global/features/multiclusterservicediscoveryresourceState:  state: ACTIVEspec: {}updateTime: '2023-10-12T06:15:31.027276757Z'

Enable multi-cluster Gateway in the fleet

Themulti-cluster GKE Gateway controllergoverns the deployment of multi-cluster Gateways.

When enabling the multi-cluster Gateway controller, you must select your configcluster. The config cluster is the GKE cluster in which yourGateway resources (Gateway,Routes,Policies) are deployed.It is a central place that controls routing across your clusters.SeeConfig cluster designto help you decide which cluster to choose as your config cluster.

1. Enable multi-cluster Gateway and specify your config cluster in your fleet.Note that you can always update the config cluster at a later time.This example specifiesgke-west-1 as the config cluster that will host theresources for multi-cluster Gateways.

   gcloudcontainerfleetingressenable\--config-membership=projects/PROJECT_ID/locations/us-west1/memberships/gke-west-1\--project=PROJECT_ID

2. Grant Identity and Access Management (IAM) permissions required by the multi-clusterGateway controller:

   gcloudprojectsadd-iam-policy-bindingPROJECT_ID\--member"serviceAccount:service-PROJECT_NUMBER@gcp-sa-multiclusteringress.iam.gserviceaccount.com"\--role"roles/container.admin"\--project=PROJECT_ID

   ReplacePROJECT_ID andPROJECT_NUMBER with the project ID and project number where yourclusters are deployed.

3. Confirm that the GKE Gateway controller is enabled for yourfleet:

   gcloudcontainerfleetingressdescribe--project=PROJECT_ID

   The output is similar to the following:

   createTime: '2023-10-12T06:23:06.732858524Z'membershipStates:  projects/441323991697/locations/us-east1/memberships/gke-east-1:    state:      code: OK      updateTime: '2023-10-12T06:30:08.815839024Z'  projects/441323991697/locations/us-west1/memberships/gke-west-1:    state:      code: OK      updateTime: '2023-10-12T06:30:08.815837031Z'  projects/441323991697/locations/us-west1/memberships/gke-west-2:    state:      code: OK      updateTime: '2023-10-12T06:30:08.815840985Z'name: projects/pierre-louis-playground/locations/global/features/multiclusteringressresourceState:  state: ACTIVEspec:  multiclusteringress:    configMembership: projects/pierre-louis-playground/locations/us-west1/memberships/gke-west-1state:  state:    code: OK    description: Ready to use    updateTime: '2023-10-12T06:23:51.317464415Z'updateTime: '2023-10-12T06:30:09.439319551Z'

4. Confirm that the GatewayClasses exist in your config cluster:

   kubectlgetgatewayclasses--context=gke-west-1

   The output is similar to the following:

   NAME                                  CONTROLLER                  ACCEPTED   AGEgke-l7-global-external-managed        networking.gke.io/gateway   True       78mgke-l7-global-external-managed-mc     networking.gke.io/gateway   True       4m22sgke-l7-gxlb                           networking.gke.io/gateway   True       78mgke-l7-gxlb-mc                        networking.gke.io/gateway   True       4m23sgke-l7-regional-external-managed      networking.gke.io/gateway   True       78mgke-l7-regional-external-managed-mc   networking.gke.io/gateway   True       4m22sgke-l7-rilb                           networking.gke.io/gateway   True       78mgke-l7-rilb-mc                        networking.gke.io/gateway   True       4m22s

   This output includes the GatewayClass gke-l7-global-external-managed-mc,gke-l7-regional-external-managed-mc, gke-l7-gxlb-mc for external multi-clusterGateways and the GatewayClass gke-l7-rilb-mc for internal multi-cluster Gateways.

5. Switch your kubectl context to the config cluster:

   kubectlconfiguse-contextgke-west-1

You are now ready to begin deploying multi-cluster Gateways in the config cluster.

Use multi-cluster Gateway with Shared VPC

A multi-cluster Gateway can also be deployed in a Shared VPC environment,with different topologies, depending on the use case.

The following table describes the supported multi-cluster Gateway topologieswithin a Shared VPC environment:

To create multi-cluster Gateways in a Shared VPC environment, use thefollowing steps:

1. Follow the steps to set up yourmulti-cluster Services with Shared VPC.

2. Create and export your services.to the config cluster

3. If you plan to use an multi-cluster internal Gateway,create a proxy-only subnet.

4. Create your multi-clusterexternalorinternalGateway and HTTPRoutes.

Once you are done with these steps, you can validate your deployment, dependingon your topology.

Troubleshooting

This section shows you how to resolve issues related to the multi-clusterGateway controller enablement.

GatewayClasses are not available in the config cluster

The following error might occur when you run the commandkubectl get gatewayclasses:

error: the server doesn't have a resource type "gatewayclasses"

To resolve this issue, install the Gateway API on your cluster:

gcloudcontainerclustersupdateCLUSTER_NAME\--gateway-api=standard\--location=CONTROL_PLANE_LOCATION

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• CONTROL_PLANE_LOCATION: the Compute Enginelocation of the control plane of yourcluster. Provide a region for regional clusters, or a zone for zonal clusters.

Multi-cluster Gateway controller doesn't start

When you enable theGateway APIon the config cluster and include the CRDsafter the Fleet ingressfeature is enabled, the multi-cluster Gateway controller mightfail to start.

You can avoid this issue by enabling the Gateway API before you enable Fleetingress. However, if you've already enabled Fleet ingress, resolve thisissue by disabling and then re-enabling Fleet ingress:

1. Disable Fleet ingress:

   gcloudcontainerfleetingressdisable

2. Enable Fleet ingress:

   gcloudcontainerfleetingressenable\--config-membership=CONFIG_MEMBERSHIP\--project=PROJECT_ID

Replace the following:

• CONFIG_MEMBERSHIP: the ID of the membership or fullyqualified identifier for the membership. For example,projects/PROJECT_ID/locations/us-west1/memberships/gke-west-1.
• PROJECT_ID: the project ID where yourGKE clusters are running.

What's next

• Learn how todeploy an external multi-cluster Gateway.
• Deploy an internal multi-cluster Gateway.
• Deploy a multi-cluster Gateway for weighted traffic splitting.
• Deploy a multi-cluster Gateway for capacity-based load balancing.