Supported features using Istio APIs (managed control plane)

This page describes the supported features and limitations forCloud Service Mesh usingTRAFFIC_DIRECTOR orISTIOD as the control plane and thedifferences between each implementation. Note that these are not options you canchoose. TheISTIOD implementation is only available for existing users.New installations use theTRAFFIC_DIRECTOR implementation when possible.

For the list of Cloud Service Mesh supported features for an in-clustercontrol plane, seeUsing Istio APIs (in-clusteristiod control plane).If you're not sure which Cloud Service Mesh control plane you're using, youcan check your control plane implementation using the instructions inIdentify control plane implementation

Limitations

Caution: All Cloud Service Mesh clusters for one mesh must be registered to the same fleet at all times to use Cloud Service Mesh. Other clusters in the project of a Cloud Service Mesh cluster must not be registered to a different fleet.

The following limitations apply:

• GKE clusters must be in one of the supportedregions.
• GKE version must be a supportedversion.
• Only the platforms listed inEnvironments are supported.
• Pods running withhostNetwork: trueare not supported.
• Managed Cloud Service Mesh must not be deployed in a cluster that also hasan in-cluster deployment of OSS Istio or an in-cluster Cloud Service Meshcontrol plane.
• Changingrelease channels isnot supported.
• Migrations frommanaged Cloud Service Mesh withasmclitoCloud Service Mesh with fleet APIare not supported. Similarly, provisioning managed Cloud Service Mesh withfleet API from--management manual to--management automatic is notsupported.
• Migrations from in-cluster Cloud Service Mesh are only supported using theCanary cluster migrationstrategy.
• Migrations and upgrades are supported only from in-cluster Cloud Service Meshversions that are in theSupported versionstable and using Mesh CA or Certificate Authority Service. Installations withIstio CA (previously known as Citadel) must firstmigrate to Mesh CA.
• The limits on scale are outlined in thisguide
• Only multi-primary deployment option for multi-cluster is supported:primary-remote deployment option for multi-cluster is not.
• istioctl ps is not supported. Instead you can use thegcloud beta container fleet mesh debug commands as described inTroubleshooting.

• Unsupported APIs:

  • EnvoyFilter API

  • WasmPlugin API

  • IstioOperator API

  • Kubernetes Ingress API

  Note: You can still enable optional features without using the IstioOperator API,seeEnable optional features on managed Cloud Service Mesh. You can also use the migration toolincluded withasmcli to automatically convert otherIstioOperatoroptional features to be compatible with managed control plane. For moreinformation, seeMigrate fromIstioOperator.

• For a list of unsupported API fields, seeUnsupported Istio APIs in Managed Cloud Service Mesh.

• You can use the managed control plane without a GKE Enterprise subscription,but certain UI elements and features in Google Cloud console are only availableto GKE Enterprise subscribers. For information about what is availableto subscribers and non-subscribers, seeGKE Enterprise and Cloud Service Mesh UI differences.

• During the provisioning process for a managed control plane,Istio CRDs corresponding to the selected channel are installed in thespecified cluster. If there are existing Istio CRDs in the cluster, they willbe overwritten.

• Managed Cloud Service Mesh only supports the default DNS domain.cluster.local.

• New installations of managed Cloud Service Mesh fetch JWKS only usingEnvoys, unless the fleet contains other clusters for which that behavior isnot enabled. This is equivalent to thePILOT_JWT_ENABLE_REMOTE_JWKS=envoyIstio option. Compared to installations that don't have VPCSC_GA_SUPPORTEDcondition (see below), you might need extra configuration forServiceEntryandDestinationRule configurations. For an example, seerequestauthn-with-se.yaml.tmpl.You can determine whether the current mode of operation is equivalent toPILOT_JWT_ENABLE_REMOTE_JWKS=envoy by determining whetherVPC ServiceControls are supported for the controlplane (ie. the VPCSC_GA_SUPPORTED condition is displayed).

• Managed data plane is only supported for workloads without additional sidecars(other than the Cloud Service Mesh sidecar).

Control plane differences

There are differences in supported features between theISTIOD andTRAFFIC_DIRECTORcontrol plane implementations. To check which implementation you are using, seeIdentify control plane implementation.

• – indicates the feature is available andenabled by default.
• † - indicates that feature APIs mayhave differences between various platforms.
• * – indicates the feature is supported forthe platform and can be enabled, as described inEnable optional featuresor the feature guide linked in the feature table.
• § – indicates that the feature issupported by allowlist. Previous users of managed Anthos Service Mesh areautomatically allowlisted at theorganization level.Contact Google Cloud Support to request accessor to check your allowlist status.
• – indicates either the feature isn'tavailable or it isn't supported.

The default and optional features are fully supported by Google CloudSupport. Features not explicitly listed in the tables receive best-effortsupport.

What determines control plane implementation

When you provision managed Cloud Service Mesh the first time in a fleet, wedetermine which control plane implementation to use. The same implementation isused for all clusters that provision managed Cloud Service Mesh in that fleet.

New fleets that onboard to managed Cloud Service Mesh receive theTRAFFIC_DIRECTOR control plane implementation, with certain exceptions:

• If you are an existing managed Cloud Service Mesh user, you receive theISTIODcontrol plane implementation when you onboard a new fleet in the same Google CloudOrganization to managed Cloud Service Mesh, until at least June 30, 2024.If you are one of these users, you can contact Support to fine-tune this behavior.Users whose existing usage is not compatible with theTRAFFIC_DIRECTORimplementation without changes will continue to receive theISTIODimplementation until September 8, 2024. (These users received a ServiceAnnouncement.)
• If any GKE on Google Cloud cluster in your fleet contains an in-cluster Cloud Service Meshcontrol plane when you provision managed Cloud Service Mesh, you willreceive theISTIOD control plane implementation.
• If any cluster in your fleet usesGKE Sandbox,when you provision managed Cloud Service Mesh, you receive theISTIODcontrol plane implementation.

Managed control plane supported features

Install, upgrade, and rollback

Feature	Managed (TD)	Managed (istiod)
Installation on GKE clusters usingfleet feature API
Upgrades from ASM 1.9 versions that use Mesh CA
Direct (skip-level) upgrades from Cloud Service Mesh versions prior to 1.9 (see notes for indirect upgrades)
Direct (skip-level) upgrades from Istio OSS (see notes for indirect upgrades)
Direct (skip-level) upgrades from Istio-on-GKE add-on (see notes for indirect upgrades)
Enablingoptional features

Note: Indirect migrations and upgrades are supported, meaning that you can followthe standard Cloud Service Meshupgrade pathsthrough each version until you reach Cloud Service Mesh 1.28 with anin-cluster control plane, then you can migrate to the managed control plane.Direct migrations and upgrades are supported only from Cloud Service Mesh versions1.9 or later installed with Mesh CA. Installations with Istio CA must firstmigrate to Mesh CA.

Environments

Feature	Managed (TD)	Managed (istiod)
GKE versionscurrently available in release channels, in one of the supportedregions
GKE versionscurrently available in release channels, in one of the supportedregions, GKEAutopilot clusters
Environments outside of Google Cloud (GKE Enterprise on-premises, GKE Enterprise on other public clouds, Amazon EKS, Microsoft AKS, or other Kubernetes clusters)

Scale

Refer to theScalability Limits page

Platform environment

Feature	Managed (TD)	Managed (istiod)
Single network
Multi-network
Single-project
Multi-project with Shared VPC

Multi-cluster deployment

Feature	Managed (TD)	Managed (istiod)
Multi-primary
Primary-remote
Multi-cluster endpoint discovery withdeclarative API
Multi-cluster endpoint discovery withremote secrets
Multi-cluster endpoint discovery withdeclarative API and a simple topology

Notes on terminology

• A multi-primary configuration means that the configuration must bereplicated in all clusters.
• A primary-remote configuration means that a single cluster contains theconfiguration and is considered the source of truth.
• Cloud Service Mesh uses a simplified definition of network based ongeneral connectivity. Workload instances are on the same network if they areable to communicate directly, without a gateway.
• Simple topology for multi-cluster endpoint discovery means that everycluster in the fleet either participates or does not participate inendpoints discovery. Complex topologies that are unsupported include (a)one-way endpoint discover (e.g. cluster A can discover endpoints in clusterB but not vice versa) and (b) disjointed endpoint discovery networks (e.g.clusters A and B can discover each other's endpoints, clusters C and D candiscover each other's endpoints, but A/B and C/D cannot discover eachother's endpoints).

Base Images

Feature	Managed (TD)	Managed (istiod)
Distroless proxy image	†

† Cloud Service Mesh with a managed (TD) control plane only supportsthe distroless image type. You cannot change it.

Note that distroless images have minimal binaries, so you cannot exec the usualcommands like bash or curl because they are not present in the distroless image.However, you can use ephemeral containers to attach to a running workload Pod tobe able to inspect it and run custom commands. For example, seeCollecting Cloud Service Mesh logs.

Security

VPC Service Controls

Feature	Managed (TD)	Managed (istiod)
VPC Service Controls

Certificate distribution and rotation mechanisms

Feature	Managed (TD)	Managed (istiod)
Workload certificate management
External certificate management on ingress andegress gateways.

Certificate authority (CA) support

Feature	Managed (TD)	Managed (istiod)
Cloud Service Mesh certificate authority
Certificate Authority Service
Istio CA
Integration with custom CAs

Security features

In addition to supporting Istio security features, Cloud Service Meshprovides even more capabilities to help you secure your applications.

Feature	Managed (TD)	Managed (istiod)
IAP integration
End-user authentication
Dry-run mode
Denial logging
Audit policies (not supported)

Authorization policy

Feature	Managed (TD)	Managed (istiod)
Authorization v1beta1 policy
CUSTOM Authorization Policy	§

Peer authentication

Feature	Managed (TD)	Managed (istiod)
Auto-mTLS
mTLS PERMISSIVE mode
mTLS STRICT mode	*	*
mTLS DISABLE mode

Request authentication

Feature	Managed (TD)	Managed (istiod)
JWT authentication(Note 1)
JWT Claim Based Routing
JWT Copy Claim to Headers

Notes:

1. Third-party JWT is enabled by default.
2. Add the full fqdn/hostname in JWKSURI while defining RequestAuthentication API.
3. Managed control plane enforces Envoy to fetch JWKS when specifying JWKS URI.
4. AES CBC algorithms A128CBC-HS256, A192CBC-HS384, and A256CBC-HS512 are not allowed in inlined JWKS when usingTRAFFIC_DIRECTOR control plane implementation.

Telemetry

Metrics

Feature	Managed (TD)	Managed (istiod)
Cloud Monitoring (HTTP in-proxy metrics)
Cloud Monitoring (TCP in-proxy metrics)
Prometheus metrics export to Grafana (Envoy metrics only)	*	*
Prometheus metrics export to Kiali
Google Cloud Managed Service for Prometheus, not including the Cloud Service Mesh dashboard	*	*
Istio Telemetry API	†
Custom adapters/backends, in or out of process
Arbitrary telemetry and logging backends

† TheTRAFFIC_DIRECTOR control plane supports a subset of Istio telemetry APIused to configureaccess logs andtrace. TheTRAFFIC_DIRECTORcontrol plane does not support configuring the trace sampling rate.

Note: You can configure third-party telemetry products (such as Grafana,and Prometheus), but we cannot guarantee future compatibility and Cloud Supportcannot provide help managing them.

Proxy request logging

Feature	Managed (TD)	Managed (istiod)
Traffic logs
Access logs	*	*

Tracing

Feature	Managed (TD)	Managed (istiod)
Cloud Trace	*	*
Jaeger tracing (allows use of customer-managed Jaeger)		Compatible
Zipkin tracing (allows use of customer-managed Zipkin)		Compatible

Note: You can configure third-party telemetry products (such as Jaeger, Zipkin,and Grafana), but we cannot guarantee future compatibility and Cloud Supportcannot provide help managing them.

Networking

Traffic interception and redirection mechanisms

Feature	Managed (TD)	Managed (istiod)
Use ofiptables usinginit containers withCAP_NET_ADMIN		†
Istio Container Network Interface (CNI)
Whitebox sidecar

† We strongly recommend using Container Network Interface (CNI) instead ofinit containers.

Protocol support

Feature	Managed (TD)	Managed (istiod)
IPv4
HTTP/1.1
HTTP/2
TCP byte streams (Note 1)
gRPC
IPv6	†

Notes:

1. Although TCP is a supported protocol for networking and TCPmetrics are collected, they are not reported. Metrics are displayed only forHTTP services in the Google Cloud console.
2. Services that are configured with Layer 7 capabilities forthe following protocols are not supported: WebSocket, MongoDB, Redis, MySQL, Kafka,Cassandra, RabbitMQ, Cloud SQL. You might be able to make the protocol work byusing TCP byte stream support. If TCP byte stream cannot support the protocol(for example, Kafka sends a redirect address in a protocol-specific reply andthis redirect is incompatible with Cloud Service Mesh's routing logic), thenthe protocol isn't supported. Although gateway ports can be created with Mongo, MySQL and Redis protocol, the mesh treats the resulting traffic as standard TCP, lacking protocol-specific handling.
3. † In proxyless gRPC, IPv6 dual-stack features are supported only in gRPC1.66.1 or newer inC++ andPython,gRPC Go v1.71, andorgRPC Node.js v1.12.If you try to configure dual-stack features with a version of gRPC that doesn'tsupport dual-stack, the clients will use only the first address sent byTraffic Director.

Envoy deployments

Feature	Managed (TD)	Managed (istiod)
Sidecars
Ingress gateway
Egress directly out from sidecars
Egress using egress gateways	*	*

CRD support

Feature	Managed (TD)	Managed (istiod)
Sidecar resource
Service entry resource
Percentage, fault injection, path matching, redirects, retries, rewriting, timeout, retry, mirroring, header manipulation, and CORS routing rules
`EnvoyFilter` API	§
`WasmPlugin` API
Istio Operator

Load balancer for the Istio ingress gateway

Feature	Managed (TD)	Managed (istiod)
Third-party external load balancer
Google Cloud Internal load balancer	*	*

Service mesh cloud gateway

Feature	Managed (TD)	Managed (istiod)
Service mesh cloud gateway

Kubernetes Gateway API

Feature	Managed (TD)	Managed (istiod)
Kubernetes Gateway API

Load balancing policies

Feature	Managed (TD)	Managed (istiod)
Round robin
Least connections
Random
Passthrough
Consistent hash
Locality
GCPTrafficDistributionPolicy
GCPBackendPolicy

Load balancing modes

Feature	Managed (TD)	Managed (istiod)
RATE
UTILIZATION
CUSTOM_METRICS
IN-FLIGHT (Preview)

For more information about balancing modes, see theBackend services overview.

Service entry

Feature	Managed (TD)	Managed (istiod)
ServiceEntry v1beta1	†

† TheTRAFFIC_DIRECTOR control plane implementation does not support followingfields and values in fields:

• workloadSelector field
• endpoints[].network field
• endpoints[].locality field
• endpoints[].weight field
• endpoints[].serviceAccount field
• DNS_ROUND_ROBIN value inresolution field
• MESH_INTERNAL value inlocation field
• Unix domain socket address inendpoints[].address field
• subjectAltNames field
• Two or moreendpoints[] entries ifresolution field hasDNS value

Destination rule

Feature	Managed (TD)	Managed (istiod)
DestinationRule v1beta1	†

† TheTRAFFIC_DIRECTOR control plane implementation does not support following fields.

• trafficPolicy.loadBalancer.localityLbSetting field
• trafficPolicy.tunnel field
• trafficPolicy.tls.credentialName field
• trafficPolicy.portLevelSettings[].tls.credentialName field

Additionally, theTRAFFIC_DIRECTOR control plane implementation requires that thedestination rule defining subsets is in the same namespace and cluster withthe Kubernetes service or ServiceEntry.

Sidecar

Feature	Managed (TD)	Managed (istiod)
Sidecar v1beta1	†

† TheTRAFFIC_DIRECTOR control plane implementation does not support followingfields and values in fields:

• ingress field
• egress.port field
• egress.bind field
• egress.captureMode field
• inboundConnectionPool field

DNS proxy

Feature	Managed (TD)	Managed (istiod)
Name resolution ofService across clusters	†
Name resolution ofServiceEntry in a cluster	†
Name resolution ofHeadlessService
Address auto allocation

†1.21.5-asm.39 or later version of the sidecar is required.

MeshConfig

Feature	Managed (TD)	Managed (istiod)
DiscoverySelectors
clusterLocal
LocalityLB	§
ExtensionProviders	§
CACert
ImageType - distroless	§
OutboundTrafficPolicy	§
defaultProviders.accessLogging
defaultProviders.tracing
defaultConfig.tracing.stackdriver	§
accessLogFile	§

ProxyConfig

Feature	Managed (TD)	Managed (istiod)
HTTP/1.0 support (ISTIO_META_NETWORK)
Image selection (distroless or base image)	†
Kubernetes native sidecar (ENABLE_NATIVE_SIDECARS)

†Distroless image is used for the injection.

Regions

GKE clusters must be in one of the following regions or any zonewithin the following regions.

Region	Location
africa-south1	Johannesburg
asia-east1	Taiwan
asia-east2	Hong Kong
asia-northeast1	Tokyo, Japan
asia-northeast2	Osaka, Japan
asia-northeast3	South Korea
asia-south1	Mumbai, India
asia-south2	Delhi, India
asia-southeast1	Singapore
asia-southeast2	Jakarta
australia-southeast1	Sydney, Australia
australia-southeast2	Melbourne, Australia
europe-central2	Poland
europe-north1	Finland
europe-north2	Stockholm
europe-southwest1	Spain
europe-west1	Belgium
europe-west2	England
europe-west3	Frankfurt, Germany
europe-west4	Netherlands
europe-west6	Switzerland
europe-west8	Milan, Italy
europe-west9	France
europe-west10	Berlin, Germany
europe-west12	Turin, Italy
me-central1	Doha
me-central2	Dammam, Saudi Arabia
me-west1	Tel Aviv
northamerica-northeast1	Montreal, Canada
northamerica-northeast2	Toronto, Canada
northamerica-south1	Mexico
southamerica-east1	Brazil
southamerica-west1	Chile
us-central1	Iowa
us-east1	South Carolina
us-east4	Northern Virginia
us-east5	Ohio
us-south1	Dallas
us-west1	Oregon
us-west2	Los Angeles
us-west3	Salt Lake City
us-west4	Las Vegas

User interface

Feature	Managed (TD)	Managed (istiod)
Cloud Service Mesh dashboards in the Google Cloud console
Cloud Monitoring
Cloud Logging

Tooling

Feature	Managed (TD)	Managed (istiod)
gcloud beta container fleet mesh debug tool