Provision a managed Cloud Service Mesh control plane on GKE

Cloud Service Mesh is a Google-managed service mesh that you simply enable.Google handles reliability, upgrades, scaling, and security for you.

This page shows you how to use thefleet API to set up managedCloud Service Mesh using Istio APIs.

Prerequisites

As a starting point, this guide assumes that you have:

• A Cloud project
• A Cloud Billing account
• Obtained therequired permissions to provision Cloud Service Mesh
• EnabledWorkload Identity on your clusters and node pools.

Requirements

• One or more clusters with asupported version of GKE,in one of the supportedregions.

• Note that managed Cloud Service Mesh uses GKE releasechannels to balance between stability and upgrade speed. New changes toCloud Service Mesh in-cluster components (including CNI, MDPC, proxies, andIstio CRDs) will rollout to clusters that subscribe GKE rapidchannel first. They will then be promoted to GKE regularchannel, and finally the GKE stable channel, once theydemonstrate enough stability.

  • Managed Cloud Service Mesh doesn't support changing GKErelease channels safely.
  • If you do change the GKE release channel,Cloud Service Mesh automatically upgrades/downgrades the in-clustercomponents (CNI, MDPC, default injected proxy version and Istio CRDs) toalign with the current GKE release channel.

• Ensure that your cluster has enough capacity for the required components thatmanaged Cloud Service Mesh installs in the cluster.

  • Themdp-controller deployment inkube-system namespace requests cpu:50m, memory: 128Mi.
  • Theistio-cni-node daemonset inkube-system namespace requests cpu: 100m,memory: 100Mi on each node.

• Ensure that the organizational policyconstraints/compute.disableInternetNetworkEndpointGroup is disabled.If the policy is enabled, ServiceEntry may not work.

• Ensure that the client machine that you provision managed Cloud Service Mesh from hasnetwork connectivity to the API server.

• Your clusters must beregistered to a fleet.This is included in the instructions if it's not already registered prior to theprovision.

• Your project must have the Service Mesh fleet feature enabled. This isincluded in the instructions if you haven't enabled it yet.

• GKE Autopilot is only supported with GKE version1.21.3+.

• Cloud Service Mesh can use multiple GKE clusters in asingle-project single-network environment or a multi-project single-networkenvironment.

  • If you join clusters that are not in the same project, they must beregistered to the samefleet host project,and the clusters must be in ashared VPCconfiguration together on the same network.
  • For a single-project multi-cluster environment, the fleet project can bethe same as the cluster project. For more information about fleets, seeFleets Overview.
  • For a multi-project environment, we recommend that you host the fleet in aseparate project from the cluster projects. If your organizationalpolicies and existing configuration allow it, we recommend that you usethe shared VPC project as the fleet host project. For more information,seeSetting up clusters with Shared VPC.

Roles required to install Cloud Service Mesh

The following table describes the roles that are required to install managedCloud Service Mesh.

Roles required to run Cloud Service Mesh

The following table describes the roles that are required by the service accountto run managed Cloud Service Mesh. If your network or cluster projects differfrom the fleet host project, you need to give the service account in the fleetproject access to these roles in the other projects.

Limitations

We recommend that you review the list ofCloud Service Mesh supported features and limitations.In particular, note the following:

• TheIstioOperator API isn't supported since its main purpose is to controlin-cluster components.

  Note: You can still enable optional features without using the IstioOperator API,seeEnabling optional features on managed Cloud Service Mesh.You can also use the migration tool included withasmcli to automaticallyconvert otherIstioOperator optional features to be compatible withmanaged control plane. For more information, seeMigrate fromIstioOperator.

• Using Certificate Authority Service (CA Service) requiresconfiguring Cloud Service Mesh per cluster,and is not supported when using thefleet-default configurationin GKE Enterprise.

• For GKE Autopilot clusters, cross-project setup is onlysupported with GKE 1.23 or later.

• For GKE Autopilot clusters, in order to adapt to theGKE Autopilot resource limit,the default proxy resource requests and limits are set to 500m CPU and 512 Mbmemory. You can override the default values usingcustom injection.

• For GKE Autopilot clusters, there may be warnings forCloud Service Mesh components "DaemonSet has no nodes selected" until theclusters' NodePool scales.

• During the provisioning process for a managed control plane, Istio CRDs areprovisioned in the specified cluster. If there are existing Istio CRDs in thecluster, they will be overwritten.

• Istio CNI and Cloud Service Mesh are not compatible with GKE Sandbox. Therefore,managed Cloud Service Mesh with theTRAFFIC_DIRECTOR implementation doesnot support clusters with GKE Sandbox enabled.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

1. Configuregcloud (even if you are using Cloud Shell).
1. Authenticate with the Google Cloud CLI, whereFLEET_PROJECT_ID is the ID of your Fleet Host project. Generally, theFLEET_PROJECT_ID is created by default and has the same name as the project.

          gcloud auth login --projectFLEET_PROJECT_ID

2. Update the components:

          gcloud components update

2. Enable the required APIs on your fleet host project.

     gcloud services enable mesh.googleapis.com \      --project=FLEET_PROJECT_ID

Configure managed Cloud Service Mesh

The steps required to provision managed Cloud Service Mesh using the fleet API dependon whether you prefer to enable bydefault for new fleet clusters orenable it per cluster.

Configure for your fleet

You must haveGoogle Kubernetes Engine (GKE) Enterprise edition enabled toenable managed Cloud Service Mesh as a default configuration for your fleet.This means that every new GKE on Google Cloud clusterregistered during cluster creationwill have managed Cloud Service Mesh enabled on the cluster. You can find outmore about fleet default configuration inManage fleet-levelfeatures.

Enabling managed Cloud Service Mesh as a default configuration for your fleet and registeringclusters to the fleet during cluster creation only supportsMesh CA. If you want to useCertificate Authority Service, we recommend that youenable it per cluster.

To enable fleet-level defaults for managed Cloud Service Mesh, complete thefollowing steps:

Proceed toVerify the control plane has been provisioned.

Configure per cluster

Use the following steps to configure managed Cloud Service Mesh for eachcluster in your mesh individually.

Enable the Cloud Service Mesh fleet feature

Enable themesh feature on the fleet project. Note that if you plan toregister multiple clusters, enabling the Cloud Service Mesh fleet featurehappens at the fleet-level so you only have to run this command once.

gcloud container fleet mesh enable --projectFLEET_PROJECT_ID

Register clusters to a fleet

1. Register a GKE clusterusing fleet workload identity. The--location flag is the compute zone orregion (such asus-central1-a orus-central1) for the cluster.

   gcloud container clusters updateCLUSTER_NAME \  --locationCLUSTER_LOCATION \  --fleet-projectFLEET_PROJECT_ID

2. Verify your cluster is registered:

   gcloud container fleet memberships list --projectFLEET_PROJECT_ID

   Example output:

   NAME                 EXTERNAL_ID                           LOCATIONcluster-1            1d8e255d-2b55-4df9-8793-0435461a2cbc  us-central1

   Make note of theMEMBERSHIP_NAME, as you will need it when youenable automatic management.

3. If your cluster's network's project differs from your fleet host project(for example you are using ashared VPC), you must allowCloud Service Mesh service accounts in the fleet project to access thenetwork project. You only need to do this once for the network project.

   Grant service accounts in the fleet project permission to access thenetwork project:

   gcloudprojectsadd-iam-policy-binding"NETWORK_PROJECT_ID"\--member"serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com"\--roleroles/anthosservicemesh.serviceAgent

4. If your cluster's project differs from your fleet host project, you must allowCloud Service Mesh service accounts in the fleet project to access the clusterproject, and enable required APIs on the cluster project.

   You only need to do this once for each cluster project. If you previously configured managed Cloud Service Mesh for this combination of cluster andfleet projects, then these changes have already been applied and you don't have to run the following commands.

   Grant service accounts in the fleet project permission to access the clusterproject:

   gcloudprojectsadd-iam-policy-binding"CLUSTER_PROJECT_ID"\--member"serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com"\--roleroles/anthosservicemesh.serviceAgent

   Enable the Mesh API on the cluster's project:

   gcloudservicesenablemesh.googleapis.com\--project=CLUSTER_PROJECT_ID

Configure Certificate Authority Service (Optional)

If your service mesh deployment requires Certificate Authority Service (CA Service),then followConfigure Certificate Authority Service for managed Cloud Service Meshto enable it for your fleet. Make sure to complete all steps before proceedingto the next section.

Enable automatic management

Run the following command to enable automatic management:

  gcloud container fleet mesh update \     --management automatic \     --membershipsMEMBERSHIP_NAME \     --projectFLEET_PROJECT_ID \     --locationMEMBERSHIP_LOCATION

Cloud Service Mesh supports provisioning via Terraform throughthe GKEHub feature membership module.

Please see theProvision managed service mesh tutorialfor detailed instructions.

Verify the control plane has been provisioned

After a few minutes, verify that the control plane status isACTIVE:

gcloud container fleet mesh describe --projectFLEET_PROJECT_ID

The output is similar to:

...membershipSpecs:projects/746296320118/locations/us-central1/memberships/demo-cluster-1:mesh:management:MANAGEMENT_AUTOMATICmembershipStates:projects/746296320118/locations/us-central1/memberships/demo-cluster-1:servicemesh:controlPlaneManagement:details:-code:REVISION_READYdetails:'Ready:asm-managed'state:ACTIVEimplementation:ISTIOD|TRAFFIC_DIRECTORdataPlaneManagement:details:-code:OKdetails:Serviceisrunning.state:ACTIVEstate:code:OKdescription:'Revision(s)readyforuse:asm-managed.'...

Take note of the control plane displayed in theimplementation field, eitherISTIOD orTRAFFIC_DIRECTOR. SeeCloud Service Mesh supported featuresfor control plane differences and supported configurations, and for how thecontrol plane implementation is selected.

Configurekubectl to point to the cluster

The following sections involve runningkubectl commands against each one ofyour clusters. Before proceeding through the following sections, run thefollowing command for each of your clusters to configurekubectl to point tothe cluster.

gcloudcontainerclustersget-credentialsCLUSTER_NAME\--locationCLUSTER_LOCATION\--projectCLUSTER_PROJECT_ID

Note that an ingress gateway isn't automatically deployed with the controlplane. Decoupling the deployment of the ingress gateway and control planelets you manage your gateways in a production environment.If you want to use an Istio ingress gateway or an egress gateway, seeDeploy gateways. If you want to usethe Kubernetes Gateway API, seePrepare Gateway for Mesh. To enableother optional features, seeEnabling optional features on Cloud Service Mesh.

Managed data plane

If you use managed Cloud Service Mesh, Google fully manages upgrades ofyour proxies.

With the managed data plane feature enabled, the sidecar proxies and injectedgateways are actively and automatically updated in conjunction with the managedcontrol plane by restarting workloads to re-inject new versions of the proxy.This starts after the control plane has been upgraded and normally completeswithin 2 weeks after starting.

Note that the managed data plane relies on the GKE releasechannel. If you change the GKE release channel while the manageddata plane is enabled, managed Cloud Service Mesh will update the proxies ofall existing workloads like an managed data plane rollout.

Ifdisabled, proxy management isdone passively - driven by the natural lifecycle of the pods in the cluster andmust be manually triggered by the user to control the update rate.

The managed data plane upgrades proxies by evicting pods that are runningearlier versions of the proxy. The evictions are done gradually, honoring thepod disruption budget and controlling the rate of change.

When new versions of the Cloud Service Mesh data plane are deployed

¹Cloud Service Mesh active updates automatically replacePodsin Workloads, except forStatefulSets,Jobs,DaemonSets, and manually injectedPods. Cloud Service Mesh active updates honorPod disruption budgets.

• Low-priority active updates coincide with GKE maintenancewindows.
• High-priority active updates can happen as soon as Cloud Service Meshmakes them available to your cluster, without respect to GKEmaintenance windows. High-priority active updates usually have at leastone associated CVE.

Enable the managed data plane if you don't want to manage the lifecycle of theCloud Service Mesh data plane yourself, and your Workloads can tolerate Podsbeing replaced at any time.

Disable the managed data plane if you'd like full control over the time of allCloud Service Mesh data plane updates.

Limitations

The managed data plane doesn't manage the following:

• Uninjected pods
• Manually injected pods
• Jobs
• StatefulSets
• DaemonSets

Disable the managed data plane (optional)

If you are provisioning managed Cloud Service Mesh on a new cluster, then you candisable the managed data plane completely, or for individual namespaces or pods.The managed data plane will continue to be disabled for existing clusters whereit was disabled by default or manually.

To disable the managed data plane at the cluster level and revert back tomanaging the sidecar proxies yourself, change the annotation:

kubectl annotate --overwrite controlplanerevision -n istio-system \  mesh.cloud.google.com/proxy='{"managed":"false"}'

To disable the managed data plane for a namespace:

kubectl annotate --overwrite namespaceNAMESPACE \  mesh.cloud.google.com/proxy='{"managed":"false"}'

To disable the managed data plane for a pod:

kubectl annotate --overwrite podPOD_NAME \  mesh.cloud.google.com/proxy='{"managed":"false"}'

Enable maintenance windows for the managed data plane

If you have a GKE maintenance window configured, activeupgrades will commence at the start of the next available maintenance window andcontinue without pause until all managed pods have been updated (usually 12hours). The maintenance window is not observed for CVE related rollouts.

Cloud Service Mesh uses theGKE maintenance windowto align with GKE.

Enable maintenance notifications for the managed data plane

You can request to be notified about upcoming managed data plane maintenance upto a week before maintenance is scheduled. Maintenance notifications are not sentby default. You must alsoConfigure a GKE maintenance windowbefore you can receive notifications. When enabled, notifications are sent atleast two days before the upgrade operation.

To opt in to managed data plane maintenance notifications:

1. Go to theCommunication page.

   Go to the Communication page

2. In theCloud Service Mesh Upgrade row, under theEmail column, select theradio button to turn maintenance notificationsON.

Each user who wants to receive notifications must opt in separately. If you wantto set an email filter for these notifications, the subject line is:

Upcoming upgrade for your Cloud Service Mesh cluster "CLUSTER_LOCATION/CLUSTER_NAME".

The following example shows a typical managed data plane maintenancenotification:

Subject Line: Upcoming upgrade for your Cloud Service Mesh cluster "<location/cluster-name>"

Dear Cloud Service Mesh user,

The Cloud Service Mesh components in your cluster ${instance_id} (https://console.cloud.google.com/kubernetes/clusters/details/${instance_id}/details?project=${project_id}) are scheduled to upgrade on ${scheduled_date_human_readable} at ${scheduled_time_human_readable}.

You can check the release notes (https://cloud.google.com/service-mesh/v1.25/docs/release-notes) to learn about the new update.

In the event that this maintenance gets canceled, you'll receive another email.

Sincerely,

The Cloud Service Mesh Team

(c) 2023 Google LLC 1600 Amphitheater Parkway, Mountain View, CA 94043You have received this announcement to update you about important changes to Google Cloud Platform or your account.You can opt out of maintenance window notifications by editing your user preferences: https://console.cloud.google.com/user-preferences/communication?project=${project_id}

Configure endpoint discovery (only for multi-cluster installations)

If your mesh has only one cluster, skip these multi-cluster steps and proceed toDeploy applications orMigrate applications.

Before you continue, ensure that Cloud Service Mesh is configured on eachcluster.

Enabling Cloud Service Mesh with the fleet API will enable endpoint discovery forthis cluster. However, you mustopen firewall ports.To disable endpoint discovery for one or more clusters, see the instructionsto disable it inEndpoint discovery between clusters with declarative API.

For an example application with two clusters, seeHelloWorld service example.

Deploy applications

Enable the namespace for injection. The steps depend on yourcontrol plane implementation.

kubectllabelnamespaceNAMESPACE\istio.io/rev-istio-injection=enabled--overwrite

kubectllabelnamespaceNAMESPACE\istio.io/rev-istio-injection=enabled--overwrite

Validate that the namespace label is applied correctly using the following command.

kubectlgetnamespace-Listio-injection

Example output:

  NAME                 STATUS   AGE     ISTIO-INJECTION  default              Active   5m9s    enabled

At this point, you have successfully configured managed Cloud Service Mesh. If youhave any existing workloads in labeled namespaces, then restart them so they getproxies injected.

If you deploy an application in a multi-cluster setup, replicate theKubernetes and control plane configuration in all clusters, unlessyou plan to limit that particular config to a subset of clusters. Theconfiguration applied to a particular cluster is the source of truth for thatcluster.

Customize injection (optional)

You can override default values and customize injection settings but this canlead to unforeseen configuration errors and resulting issues with sidecarcontainers. Before you customize injection, read the information after thesample for notes on particular settings and recommendations.

Per-pod configuration is available to override these options on individual pods.This is done by adding anistio-proxy container to your pod. The sidecarinjection will treat any configuration defined here as an override to thedefault injection template.

For example, the following configuration customizes a variety of settings,including lowering the CPU requests, adding a volume mount, and adding apreStop hook:

apiVersion:v1kind:Podmetadata:name:examplespec:containers:-name:helloimage:alpine-name:istio-proxyimage:autoresources:requests:cpu:"200m"memory:"256Mi"limits:cpu:"200m"memory:"256Mi"volumeMounts:-mountPath:/etc/certsname:certslifecycle:preStop:exec:command:["sleep","10"]volumes:-name:certssecret:secretName:istio-certs

In general, any field in a pod can be set. However, care must be taken forcertain fields:

• Kubernetes requires theimage field to be set before the injection has run.While you can set a specific image to override the default one, we recommendthat you set theimage toauto, which will cause the sidecar injectorto automatically select the image to use.
• Some fields incontainers are dependent on related settings. For example,must be less than or equal to the CPU limit. If both fields are not properlyconfigured, the pod may fail to start.
• Kubernetes lets you set bothrequests andlimits for resources in yourPodspec.GKE Autopilot only considersrequests. For moreinformation, seeSetting resource limits in Autopilot.

Additionally, certain fields are configurable by annotations on the Pod,although it is recommended to use the above approach to customizing settings.Take additional care for the following annotations:

• For GKE Standard, ifsidecar.istio.io/proxyCPU is set, makesure to explicitly setsidecar.istio.io/proxyCPULimit. Otherwise the sidecar'sCPU limit will be set as unlimited.
• For GKE Standard, ifsidecar.istio.io/proxyMemory is set,make sure to explicitly setsidecar.istio.io/proxyMemoryLimit. Otherwise thesidecar's memory limit will be set as unlimited.
• For GKE Autopilot, configuring resourcerequests andlimitsusing annotations might overprovision resources. Use the image template approachto avoid. SeeResource modification examples in Autopilot.

For example, see the below resources annotation:

spec:template:metadata:annotations:sidecar.istio.io/proxyCPU:"200m"sidecar.istio.io/proxyCPULimit:"200m"sidecar.istio.io/proxyMemory:"256Mi"sidecar.istio.io/proxyMemoryLimit:"256Mi"

Migrate applications to managed Cloud Service Mesh

1. Replace the current namespace label. The steps depend on yourcontrol plane implementation.

kubectllabelnamespaceNAMESPACE\istio.io/rev-istio-injection=enabled--overwrite

kubectllabelnamespaceNAMESPACE\istio.io/rev-istio-injection=enabled--overwrite

1. Perform a rolling upgrade of deployments in the namespace:

   kubectl rollout restart deployment -nNAMESPACE

2. Test your application to verify that the workloads function correctly.

3. If you have workloads in other namespaces, repeat the previous steps foreach namespace.

4. If you deployed the application in a multi-cluster setup, replicate theKubernetes and Istio configuration in all clusters, unless there is adesire to limit that configuration to a subset of clusters only. Theconfiguration applied to a particular cluster is the source of truth forthat cluster.

If you are satisfied that your application works as expected, you can remove thein-clusteristiod after you switch all namespaces to the managed controlplane, or keep them as a backup -istiod will automatically scale down to usefewer resources. To remove, skip toDelete old control plane.

If you encounter problems, you can identify and resolve them by usingthe information inResolving managed control plane issuesand if necessary, roll back to the previous version.

kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true

kubectl -n istio-system rollout undo deploy istio-ingressgateway

deployment.apps/istio-ingressgateway rolled back

Uninstall Cloud Service Mesh

Managed control plane auto-scales to zero when no namespaces are using it. Fordetailed steps, seeUninstall Cloud Service Mesh.