Multi Cluster Ingress is a cloud-hosted controller forGoogle Kubernetes Engine (GKE) clusters. It's aGoogle-hosted service that supports deploying shared load balancing resourcesacross clusters and across regions. To deploy Multi Cluster Ingress across multipleclusters, completeSetting up Multi Cluster Ingressthen seeDeploying Ingress across multiple clusters.

For a detailed comparison between Multi Cluster Ingress (MCI), Multi-cluster Gateway(MCG), and load balancer with Standalone Network Endpoint Groups (LB and StandaloneNEGs), seeChoose your multi-cluster load balancing API forGKE.

Multi-cluster networking

Many factors drive multi-cluster topologies, including close user proximityfor apps, cluster and regional high availability, security and organizationalseparation, cluster migration, and data locality. These use cases arerarely isolated. As the reasons for multiple clusters grow, the need for aformal and productized multi-cluster platform becomes more urgent.

Multi Cluster Ingress is designed to meet the load balancing needs of multi-cluster,multi-regional environments. It's a controller for the external HTTP(S)load balancer to provide ingress for traffic coming from the internet across oneor more clusters.

Multi Cluster Ingress's multi-cluster support satisfies many use cases including:

• A single, consistent virtual IP (VIP) for an app, independent of where the appis deployed globally.
• Multi-regional, multi-cluster availability through health checking and trafficfailover.
• Proximity-based routing through public Anycast VIPs for low client latency.
• Transparent cluster migration for upgrades or cluster rebuilds.

Default quotas

Multi Cluster Ingress has the following default quotas:

• For details about member limits for fleets, seefleet management quotas.for how many members are supported in a fleet.
• 100MultiClusterIngress resources and 100MultiClusterService resourcesper project. You can create up to 100MultiClusterIngress and 100MultiClusterService resources in a config cluster for any number of backendclusters up to the per-project cluster maximum.

Pricing and trials

To learn about Multi Cluster Ingress pricing, seeMulti Cluster Ingress pricing.

How Multi Cluster Ingress works

Multi Cluster Ingress builds on the architecture of theglobal external Application Load Balancer. Theglobal external Application Load Balancer is a globally distributed load balancer with proxiesdeployed at 100+ Google points of presence (PoPs) around the world. Theseproxies, calledGoogle Front Ends (GFEs),sit at the edge of Google's network, positioned close to clients. Multi Cluster Ingresscreates external Application Load Balancers in thePremium Tier. These load balancersuse global external IP addresses advertised usinganycast. Requests areserved by GFEs and the cluster that is closest to the client. Internet trafficgoes to the closest Google PoP and uses the Google backbone to get to aGKE cluster. This load balancing configuration results inlower latencyfrom the client to the GFE. You can also reduce latency between servingGKE clusters and GFEs by running your GKE clustersin regions that are closest to your clients.

Terminating HTTP and HTTPS connections at the edge allows the Google loadbalancer to decide where to route traffic by determining backend availabilitybefore traffic enters a data center or region. This gives traffic the mostefficient path from the client to the backend while considering the backends'health and capacity.

Multi Cluster Ingress is an Ingress controller that programs the external HTTP(S)load balancer usingnetwork endpoint groups (NEGs).When you create aMultiClusterIngress resource, GKE deploysCompute Engine load balancer resources and configures the appropriatePods across clusters as backends. The NEGs are used to track Podendpoints dynamically so the Google load balancer has the right set of healthybackends.

As you deploy applications across clusters in GKE,Multi Cluster Ingress ensures that the load balancer is in sync with events thatoccur in the cluster:

• A Deployment is created with the right matching labels.
• A Pod's process dies and fails its health check.
• A cluster is removed from the pool of backends.

Multi Cluster Ingress updates the load balancer, keeping it consistent with theenvironment and desired state of Kubernetes resources.

Multi Cluster Ingress architecture

Multi Cluster Ingress uses a centralized Kubernetes API server to deploy Ingress acrossmultiple clusters. This centralized API server is called the config cluster. AnyGKE cluster can act as the config cluster. The config clusteruses two custom resource types:MultiClusterIngress andMultiClusterService.By deploying these resources on the config cluster, the Multi Cluster Ingresscontroller deploys load balancers across multiple clusters.

The following concepts and components make up Multi Cluster Ingress:

• Multi Cluster Ingress controller - This is a globally distributedcontrol plane that runs as a service outside of your clusters. This allows thelifecycle and operations of the controller to be independent ofGKE clusters.

• Config cluster - This is a chosen GKE cluster running onGoogle Cloud where theMultiClusterIngress andMultiClusterServiceresources are deployed. This is a centralized point of control for thesemulti-cluster resources. These multi-cluster resources exist in and areaccessible from a single logical API to retain consistency across allclusters. The Ingress controller watches the config cluster and reconcilesthe load balancing infrastructure.

• Afleet lets you logically group and normalize GKE clusters, making administration of infrastructure easier and enabling the use of multi-cluster features such as Multi Cluster Ingress. You can learn more about the benefits of fleets and how to create them in thefleet management documentation. A cluster can only be a member of a single fleet.

• Member cluster - Clusters registered to a fleet are calledmember clusters.Member clusters in the fleet comprise the full scope of backends thatMulti Cluster Ingress is aware of. TheGoogle Kubernetes Engine cluster management viewprovides a secure console to view the state of all your registeredclusters.

Deployment workflow

The following steps illustrate a high-level workflow for using Multi Cluster Ingressacross multiple clusters.

1. Register GKE clusters to a fleet in your chosen project.

2. Configure a GKE cluster as the central config cluster.This cluster can be a dedicated control plane, or it can run other workloads.

3. Deploy applications to the GKE clusters where they need torun.

4. Deploy one or moreMultiClusterService resources in the config cluster withlabel and cluster matches to select clusters, namespace, and Pods that areconsidered backends for a given Service. This creates NEGs inCompute Engine, which begins to register and manage service endpoints.

5. Deploy aMultiClusterIngress resource in the config cluster thatreferences one or moreMultiClusterService resources as backends for theload balancer. This deploys the Compute Engine external load balancerresources and exposes the endpoints across clusters through a single loadbalancer VIP.

Ingress concepts

Multi Cluster Ingress uses a centralized Kubernetes API server to deploy Ingressacross multiple clusters. The following sections describe the Multi Cluster Ingressresource model, how to deploy Ingress, and concepts important for managing thishighly available network control plane.

MultiClusterService resources

AMultiClusterService is a custom resource used by Multi Cluster Ingressto represent sharing services across clusters. AMultiClusterServiceresource selects Pods, similar to the Service resource, but aMultiClusterService can also select labels and clusters. The pool of clusters that aMultiClusterService selects across are calledmember clusters. All of theclusters registered to the fleet are member clusters.

AMultiClusterService only exists in the config cluster and does not routeanything like a ClusterIP, LoadBalancer, or NodePort Service does. Instead, itlets the Multi Cluster Ingress controller refer to a singular distributed resource.

The following sample manifest describes aMultiClusterService for anapplication namedfoo:

apiVersion:networking.gke.io/v1kind:MultiClusterServicemetadata:name:foonamespace:bluespec:template:spec:selector:app:fooports:-name:webprotocol:TCPport:80targetPort:80

This manifest deploys a Service in all member clusters with the selectorapp:foo. Ifapp: foo Pods exist in that cluster, then those Pod IP addresses areadded as backends for theMultiClusterIngress.

The followingmci-zone1-svc-j726y6p1lilewtu7 is a derived Servicegenerated in one of the target clusters. This Service creates a NEG which tracksPod endpoints for all pods that match the specified label selector in thiscluster. A derived Service and NEG will exist in every target cluster for everyMultiClusterService (unless using cluster selectors). If no matching Podsexist in a target cluster then the Service and NEG will be empty. The derivedServices are managed fully by theMultiClusterService and are not managed byusers directly.

apiVersion:v1kind:Servicemetadata:annotations:cloud.google.com/neg:'{"exposed_ports":{"8080":{}}}'cloud.google.com/neg-status:'{"network_endpoint_groups":{"8080":"k8s1-a6b112b6-default-mci-zone1-svc-j726y6p1lilewt-808-e86163b5"},"zones":["us-central1-a"]}'networking.gke.io/multiclusterservice-parent:'{"Namespace":"default","Name":"zone1"}'name:mci-zone1-svc-j726y6p1lilewtu7namespace:bluespec:selector:app:fooports:-name:webprotocol:TCPport:80targetPort:80

A few notes about the derived Service:

• Its function is as a logical grouping of endpoints as backends for Multi Cluster Ingress.
• It manages the lifecycle of the NEG for a given cluster and application.
• It's created as a headless Service. Note that only theSelector andPortsfields are carried over from theMultiClusterService to the derived service.
• The Ingress controller manages its lifecycle.

MultiClusterIngress resource

AMultiClusterIngress resource behaves identically in many ways to thecore Ingress resource. Both have the same specification for defining hosts,paths, protocol termination and backends.

The following manifest describes aMultiClusterIngress that routes traffic tothefoo andbar backends depending on the HTTP host headers:

apiVersion:networking.gke.io/v1kind:MultiClusterIngressmetadata:name:foobar-ingressnamespace:bluespec:template:spec:backend:serviceName:default-backendservicePort:80rules:-host:foo.example.combackend:serviceName:fooservicePort:80-host:bar.example.combackend:serviceName:barservicePort:80

ThisMultiClusterIngress resource matches traffic to the virtual IP address onfoo.example.com andbar.example.com by sending this traffic to theMultiClusterService resources namedfoo andbar. ThisMultiClusterIngresshas a default backend that matches on all other traffic and sends that trafficto the default-backendMultiClusterService.

The following diagram shows how traffic flows from an Ingress to a cluster:

In the diagram, there are two clusters,gke-us andgke-eu. Traffic flowsfromfoo.example.com to the Pods that have the labelapp:foo across bothclusters. Frombar.example.com, traffic flows to the Pods that have the labelapp:bar across both clusters.

Ingress resources across clusters

The config cluster is the only cluster that can haveMultiClusterIngress andMultiClusterService resources. Each target cluster which has Pods that matchtheMultiClusterService label selectors also have a corresponding derivedService scheduled on them. If a cluster is explicitly not selected by aMultiClusterService, then a corresponding derived Service is not created inthat cluster.

Namespace sameness

Namespace sameness is a property of Kubernetes clusters in which a namespaceextends across clusters and is considered the same namespace.

In the following diagram, namespaceblue exists across thegke-cfg,gke-eu andgke-us GKE clusters. Namespace sameness considersthe namespaceblue to be the same across all clusters. This means that a userhas the same privileges to resources in theblue namespace in every cluster.Namespace sameness also means that Service resources with the same name acrossmultiple clusters in namespaceblue are considered to be the same Service.

The Gateway treats the Service as a single pool of endpoints across the threeclusters. Because Routes andMultiClusterIngress resources can only route toServices within the same namespace, this provides consistent multi-tenancy forconfig across all clusters in the fleet. Fleets provide a high degree ofportability since resources can be deployed or moved across clusters without anychanges to their config. Deployment into the same fleet namespace providesconsistency across clusters.

Consider the following design principles for Namespace sameness:

• Namespaces for different purposes should not have the same name acrossclusters.
• Namespaces should be reserved either explicitly, by allocating anamespace, or implicitly, through out-of-band policies, for teams andclusters within a fleet.
• Namespaces for the same purpose across clusters should share the same name.
• User permission to Namespaces across clusters should be tightlycontrolled to prevent unauthorized access.
• You should not use the default Namespace or generic Namespaces like"prod" or "dev" for normal application deployment. It is too easy for usersto deploy resources to the default Namespace accidentally and violate thesegmentation principles of Namespaces.
• The same Namespace should be created across clusters wherever a giventeam or group of users must deploy resources.

Config cluster design

The Multi Cluster Ingress config cluster is a single GKE cluster whichhosts theMultiClusterIngress andMultiClusterService resources and acts asthe single control point for Ingressacross the fleet of target GKE clusters. You choose theconfig cluster when you enable Multi Cluster Ingress. You can choose anyGKE cluster as the config cluster and change the configcluster at any time.

Config cluster availability

Because the config cluster is a single point of control, Multi Cluster Ingress resourcescannot be created or updated if the config cluster API is unavailable. Loadbalancers and traffic being served by them will not be affected by a configcluster outage but changes toMultiClusterIngress andMultiClusterServiceresources will not be reconciled by the controller until it is available again.

Consider the following design principles for config clusters:

• The config cluster should be chosen such that it is highly available.Regional clusters are preferred over zonal clusters.
• To enable Multi Cluster Ingress, the config cluster does not have to bea dedicated cluster. The config cluster may host administrative or evenapplication workloads, though you should ensure that hosted applications donot impact the availability of the config cluster API server. The configcluster can be a target cluster that hosts backends forMultiClusterServiceresources, though if extra precautions are needed the config cluster can alsobe excluded as a backend through cluster selection.
• Config clusters should have all the Namespaces that are used by targetcluster backends. AMultiClusterService resource can only reference Pods inthe same Namespace across clusters so that Namespace must be present in theconfig cluster.
• Users that deploy Ingress across multiple clusters must have access tothe config cluster to deployMultiClusterIngress andMultiClusterServiceresources. However, users should only have access to Namespaces they havepermission to use.

Selecting and migrating the config cluster

You must choose the config cluster when you enable Multi Cluster Ingress. Any membercluster of a fleet can be selected as the config cluster. You can update the configcluster at any time but must take care to ensure that it does not causedisruption. The Ingress controller will reconcile whatever resources exist inthe config cluster. When migrating the config clusterfrom the current one to the next one,MultiClusterIngress andMultiClusterService resources must be identical.If the resources are not identical, the Compute Engine Load Balancersmay be updated or destroyed after the config cluster update.

The following diagram shows how a centralized CI/CD system appliesMultiClusterIngress andMultiClusterService resources to theGKE API server for the config cluster (gke-us)and a backup cluster (gke-eu) at all times so that the resources are identicalacross the two clusters. You can change the config cluster for emergencies orplanned downtime at any time without any impact because theMultiClusterIngressandMultiClusterService resources are identical.

Cluster selection

MultiClusterService resources can select across clusters. By default, thecontroller schedules a derived Service on every target cluster. If you do notwant a derived Service on every target cluster, you can define a list ofclusters using thespec.clusters field in theMultiClusterService manifest.

You might want to define a list of clusters if you need to:

• Isolate the config cluster to preventMultiClusterService resources fromselecting across the config cluster.
• Control traffic between clusters for application migration.
• Route to application backends that only exist in a subset of clusters.
• Use a single HTTP(S) virtual IP address for routing to backends thatlive on different clusters.

You must ensure that member clusters within the same fleet and regionhave unique names to prevent naming collisions.

To learn how to configure cluster selection, seeSetting up Multi Cluster Ingress.

The following manifest describes aMultiClusterService that has aclustersfield that referenceseurope-west1-c/gke-eu andasia-northeast1-a/gke-asia:

apiVersion:networking.gke.io/v1kind:MultiClusterServicemetadata:name:foonamespace:bluespec:template:spec:selector:app:fooports:-name:webprotocol:TCPport:80targetPort:80clusters:-link:"europe-west1-c/gke-eu"-link:"asia-northeast1-a/gke-asia-1"

This manifest specifies that Pods with the matching labels in thegke-asiaandgke-eu clusters can be included as backends for theMultiClusterIngress.Any other clusters are excluded even if they have Pods with theapp: foolabel.

The following diagram shows an exampleMultiClusterService configuration usingthe preceding manifest:

In the diagram, there are three clusters:gke-eu,gke-asia-1, andgke-asia-2. Thegke-asia-2 cluster is not included as a backend, even thoughthere are Pods with matching labels, because the cluster is not included in themanifestspec.clusters list. The cluster does not receive traffic formaintenance or other operations.

What's next

• Learn how toSetup Multi Cluster Ingress.
• Learn aboutDeploying multi-cluster Gateways.
• Learn aboutDeploying Multi Cluster Ingress.
• ImplementMulti Cluster Ingress for external load balancing.