This page provides a deeper dive into how fleets help you manage multi-cluster deployments, including some key fleet terminology and concepts.Fleets are a Google Cloud concept for logically organizing clusters and otherresources, letting you use and manage multi-cluster capabilities and applyconsistent policies across your systems. Fleets form a crucial part of howmulti-cluster functionality works in Google Cloud.

This guide is designed for technical readers, including system architects,platform operators, and service operators, who want to leverage multipleclusters and related infrastructure. These concepts are useful wherever yourorganization happens to be running multiple clusters, whether inGoogle Cloud, across multiple cloud providers, or on-premises.

Before reading this page, ensure that you're familiar with basic Kubernetes concepts such as clusters andnamespaces; if you're not, seeKubernetesbasics, theGKE documentation,andPreparing an application for Cloud Service Mesh.

Terminology

The following are some important terms we use when talking about fleets.

Fleet-aware resources

Fleet-aware resources are Google Cloud project resources that can be logicallygrouped and managed as fleets. Only Kubernetes clusters can currently befleet members.

Fleet host project

The implementation of fleets, like many other Google Cloud resources, isrooted in a Google Cloud project, which we refer to as the fleet host project. Agiven Google Cloud project can only have asingle fleet (or no fleets)associated with it. This restriction reinforces using Google Cloud projects toprovide stronger isolation between resources that are not governed or consumedtogether.

Grouping infrastructure

The first important concept of fleets is the concept ofgrouping—that is,choosing which pieces ofrelatedfleet-aware resourcesshould be made part of a fleet. The decision about what to grouptogether requires answering the following questions:

• Are the resources related to one another?
  • Resources that have large amounts of cross-service communication benefitthe most from being managed together in a fleet.
  • Resources in the same deployment environment (for example, your productionenvironment) should be managed together in a fleet.
• Who administers the resources?
  • Having unified (or at least mutually trusted) control over the resources iscrucial to ensuring the integrity of the fleet.

To illustrate this point, consider an organization that has multiple lines ofbusiness (LOBs). In this case, services rarely communicate across LOBboundaries, services in different LOBs are managed differently (for example,upgrade cycles differ between LOBs), and they might even have a different set ofadministrators for each LOB. In this case, it might make sense to have fleetsper LOB. Each LOB also likely adopts multiple fleets to separate theirproduction and non-production services.

As other fleet concepts are explored in the following sections, you might findother reasons to create multiple fleets as you consider your specificorganizational needs.

Sameness

An important concept in fleets is the concept of sameness. This means that when you use certain fleet-enabled features, some Kubernetes objects such as namespaces that have the same name in different clusters are treated as if they were the same thing. This normalization is done to make administering fleet resources easier. If you're using features that leverage sameness, this assumption of sameness provides some strong guidance about how to set up namespaces, services, and identities. However, it also follows what we find most organizations already implementing themselves.

Different types of sameness provide different benefits, as shown in the following table:

Sameness property	Lets you...
A namespace is considered the same across multiple clusters.	Grant access to users across the clusters. Aggregate metrics and logs across the clusters.
A combination of namespace and service name is considered the same across multiple clusters. Same-named services in the same namespace use the same label selector.	Route traffic within and across the clusters using Cloud Service Mesh. Use GKE multi-cluster Services to automatically create services in multiple clusters. Use Multi Cluster Ingress and multi-cluster Gateway to target multi-cluster Services.
A combination of namespace and service account (identity) is considered the same across multiple clusters.	Write IAM policies once for a set of workloads running on the clusters. Write Cloud Service Mesh authorization policies once for a set of workloads running on the clusters. Authenticate to peers in a service mesh usingSPIFFE certificates without distinction among workloads in the clusters.

As this suggests, different fleet features rely on different types of sameness. A smaller number of features don't use sameness at all. You can learn more about this inWhich features use sameness?, including which features you can use without having to consider fleet-wide sameness and which features may require more careful planning.

Namespace sameness

The fundamental example of sameness in a fleet is namespace sameness.Namespaces with the same name in different clusters are considered the sameby many fleet features. Another way to think about this property is that a namespaceis logically defined across an entire fleet, even if the instantiation of thenamespace exists only in a subset of the fleet resources.

Consider the followingbackend namespace example. Although the namespace isinstantiated only in Clusters A and B, it is implicitly reserved in Cluster C(it allows a service in thebackend namespace to also be scheduled into Cluster C if necessary).This means that namespaces are allocated for the entire fleet and not percluster. As such, namespace sameness requires consistent namespace ownershipacross the fleet.

Note: If you require more granular control over namespaces in your fleet, fleet team management features let you explicitly create namespaces that are defined across a subset of fleet member clusters, known as ateam scope, rather than across the entire fleet. You can find out more about this inManage teams for your fleet.

Service sameness

Cloud Service Mesh and Multi Cluster Ingress use the concept of sameness of serviceswithin a namespace. Like namespace sameness, this implies that services with thesame namespace and service name are considered to be the same service.

The service endpoints can be merged across the mesh in the case of Cloud Service Mesh.With Multi Cluster Ingress, aMultiClusterService (MCS) resource makes theendpoint merging more explicit; however, we recommend similar practices withrespect to naming. Because of this, it's important to ensure that identicallynamed services within the same namespace are actually the same thing.

In the following example, internet traffic is load balanced across a same-namedservice in thefrontend namespace present in both Clusters B and C. Similarly,using the service mesh properties within the fleet, the service in thefrontend namespace canreach a same-named service in theauth namespace present in Clusters A and C.

Identity sameness when accessing external resources

With fleet Workload Identity Federation, services within a fleet can leverage a common identity as they egress toaccess external resources such as Google Cloud services, object stores,and so on. This common identity makes it possible to give the services within afleet access to an external resource once rather than cluster-by-cluster.

To illustrate this point further, consider the following example. Clusters A, B,and C are enrolled in common identity within their fleet. When services in thebackend namespace access Google Cloud resources, their identities aremapped to a common Google Cloud service account calledback. TheGoogle Cloud service accountback can be authorized on any number ofmanaged services, from Cloud Storage to Cloud SQL. As new fleet resourcessuch as clusters are added in thebackend namespace, they automaticallyinherit the workload identity sameness properties.

Because of identity sameness, it is important that all resources in a fleetare trusted and well-governed. Revisiting the previous example, if Cluster C isowned by a separate, untrusted team, they too can create abackend namespaceand access managed services as if they were thebackend in Cluster Aor B.

Identity sameness within a fleet

Within the fleet, identity sameness is used similarly to the external identitysameness we previously discussed. Just as fleet services are authorized oncefor an external service, they can be authorized internally as well.

In the following example, we are usingCloud Service Meshto create a multi-cluster service mesh wherefrontend has access tobackend.With Cloud Service Mesh andfleets, we don't need to specify thatfrontend in clusters B and C canaccessbackend in Clusters A and B. Instead, we just specify thatfrontendin the fleet can accessbackend in the fleet. This property not only makesauthorization simpler, it also makes the resource boundaries more flexible; nowworkloads can easily be moved from cluster to cluster without affecting how theyare authorized. As with workload identity sameness, governance over the fleetresources is crucial to ensuring the integrity of service-to-servicecommunication.

Which features use sameness?

A number of fleet features don't rely on sameness at all, and can be enabled and used without having to consider whether you want to assume any form of sameness across your fleet. Other features (including Config Sync and Policy Controller)can use sameness - for example, if you want to select a namespace across multiple fleet member clusters for configuration from a single source of truth - but do not require it for all use cases. Finally, there are features such as Multi Cluster Ingress and fleet-wide Workload Identity Federation that always assume some form of sameness across clusters, and may have to be adopted with care depending on your needs and existing workloads.

Some fleet features (such as fleet Workload Identity Federation) require that your entire fleet is ready for the assumptions of sameness that they use. Other features such as team management let you gradually opt in to sameness at the namespace orteam scope level.

The following table shows which featuresrequire one or more of the sameness concepts described in this section.

Feature	Supports gradual sameness adoption	Depends on namespace sameness	Depends on service sameness	Depends on identity sameness
Fleets	N/A	No	No	No
Binary Authorization	N/A	No	No	No
Inter-node transparent encryption	N/A	No	No	No
Fully Qualified Domain Name-based Network Policy	N/A	No	No	No
Connect gateway	N/A	No	No	No
Config Sync	N/A	No	No	No
Policy Controller	N/A	No	No	No
GKE Security Posture	N/A	No	No	No
Advanced Vulnerability Insights	N/A	No	No	No
Compliance Posture	N/A	No	No	No
Rollout sequencing	N/A	No	No	No
Team management	Yes	Yes	Yes	No
Multi Cluster Ingress	Yes	Yes	Yes	Yes
Multi Cluster Services	Yes	Yes	Yes	Yes
Fleet Workload Identity Federation	No	Yes	Yes	Yes
Cloud Service Mesh	No	Yes	Yes	Yes

Exclusivity

Fleet-aware resources can only be members of a single fleet at any giventime, a restriction that is enforced by Google Cloud tools andcomponents. This restriction ensures that there is only one source of truthgoverning a cluster. Without exclusivity, even the most simple components wouldbecome complex to use, requiring your organization to reason about and configurehow multiple components from multiple fleets would interact.

High trust

Service sameness, workload identity sameness, and mesh identity sameness arebuilt on top of a principle of high trust between members of a fleet. Thistrust makes it possible to uplevel management of these resources to the fleet,rather than managing resource-by-resource (that is, cluster-by-cluster forKubernetes resources), and ultimately makes the cluster boundary less important.

Put another way, within a fleet, clusters provide protection from blastradius concerns, availability (of both the control plane and underlyinginfrastructure), noisy neighbors, and so on. However, they are not a strongisolation boundary for policy and governance because administrators of anymember in a fleet can potentially affect the operations of services in othermembers of the fleet.

For this reason, we recommend that clusters that are not trusted by the fleetadministrator be placed in their own fleets to keep them isolated. Then, asnecessary, individual services can be authorized across the fleet boundary.

Team scopes

A team scope is a mechanism for further subdividing your fleet into groups ofclusters, letting you define the fleet-aware resourcesassigned to a specificapplication team. Depending on your use case, an individual fleet member cluster can be associated with no teams,one team, or multiple teams, allowing multiple teams to share clusters. You can also use team scopes tosequencecluster upgrade rolloutsacross your fleet, although this requires that each cluster is only associated with a singleteam.

A team scope can have explicitly defined fleet namespaces associated with it, where the namespace is considered the same across thescope.This gives you more granular control over namespaces than the defaultnamespace sameness provided by fleets alone.

Fleet-enabled components

The following components all leveragefleet concepts such as namespace and identity sameness to provide a simplifiedway to work with your clusters and services. For any current requirements orlimitations for using fleets with each component, see thecomponentrequirements.

• Workload identity pools
  A fleet offers a commonworkload identity pool that can be used toauthenticate and authorize workloads uniformly within a service mesh and toexternal services.

• Cloud Service Mesh
  Cloud Service Mesh is a suite of tools that helps you monitor and manage a reliableservice mesh on Google Cloud, on-premises, and othersupported environments.You can form a service mesh across clustersthat are part of the same fleet.

• Config Sync
  Config Sync lets you deploy and monitor declarativeconfiguration packages for your system stored in a central source of truth, like a Git repository,leveraging core Kubernetes concepts such as namespaces, labels, and annotations.With Config Sync, configuration are defined acrossthe fleet, but applied and enforced locally in each of the member resources.

• Policy Controller
  Policy Controller lets you apply and enforce declarative policies foryour Kubernetes clusters. These policies act as guardrails and can help withbest practices, security, and compliance management of your clusters and fleet.

• Multi Cluster Ingress
  Multi Cluster Ingress uses the fleet to define the set of clusters and serviceendpoints that traffic can be load balanced over, enabling low-latency andhigh-availability services.

What's next?

• Read more about when to use multiple clusters to meet your technical and business needs inMulti-cluster use cases.
• Ready to think about applying these concepts to your own systems? SeePlan fleet resources.