Set up a multi-cluster mesh outside Google Cloud

This guide explains how to set up a multi-cluster mesh for the followingplatforms:

• Google Distributed Cloud (software only) for VMware
• Google Distributed Cloud (software only) for bare metal
• GKE on Azure (deprecated)
• GKE on AWS (deprecated)
• Attached clusters, including Amazon EKS clusters and Microsoft AKS clusters (deprecated)

This guide shows how to set up two clusters, but you can extend this process toincorporate any number of clusters into your mesh.

Before you begin

This guide assumes you installed Cloud Service Mesh usingasmcli install. You needasmcli and the configuration package thatasmcli downloads to thedirectory that you specified in--output_dir when you ranasmcli install.If need to get set up, follow the steps inInstall dependent tools and validate clusterto:

• Install required tools
• Downloadasmcli
• Grant cluster admin permissions
• Validate your project and cluster

You need access to the kubeconfig files for all the clusters that you aresetting up in the mesh.

1. Create an environment variable for the project number. In the followingcommand, replaceFLEET_PROJECT_ID with thethe project ID of thefleet host project.

   exportPROJECT_NUMBER=$(gcloudprojectsdescribeFLEET_PROJECT_ID\--format="value(projectNumber)")

2. Create an environment variable for the mesh identifier.

   exportMESH_ID="proj-${PROJECT_NUMBER}"

3. Create environment variables for the cluster names in the format thatasmcli requires.

   exportCLUSTER_1="cn-FLEET_PROJECT_ID-global-CLUSTER_NAME_1"exportCLUSTER_2="cn-FLEET_PROJECT_ID-global-CLUSTER_NAME_2"

4. Get the context name for the clusters by usingthe values under theNAME column in the output of this command:

   kubectl config get-contexts

5. Set the environment variables to the cluster context names, which this guideuses in many steps later:

   export CTX_1=CLUSTER1_CONTEXT_NAMEexport CTX_2=CLUSTER2_CONTEXT_NAME

Install the east-west gateway

In the following commands:

• ReplaceCLUSTER_NAME_1 andCLUSTER_NAME_2 with the names of your clusters.

• ReplacePATH_TO_KUBECONFIG_1 andPATH_TO_KUBECONFIG_2 with the kubeconfig files foryour clusters.

Exposing services

Since the clusters are on separate networks, you need to expose all services(*.local) on the east-west gateway in both clusters. While this gateway ispublic on the Internet, services behind it can only be accessed by services witha trusted mTLS certificate and workload ID, just as if they were on the samenetwork.

1. Expose services via the east-west gateway forCLUSTER_NAME_1.

   kubectl --kubeconfig=PATH_TO_KUBECONFIG_1 apply -n istio-system -f \    asm/istio/expansion/expose-services.yaml

2. Expose services via the east-west gateway forCLUSTER_NAME_2.

   kubectl --kubeconfig=PATH_TO_KUBECONFIG_2 apply -n istio-system -f \    asm/istio/expansion/expose-services.yaml

Enable endpoint discovery

Run theasmcli create-mesh command to enable endpoint discovery. Thisexample only shows two clusters, but you can run the command to enableendpoint discovery on additional clusters, subject to theGKE Hub service limit.

./asmclicreate-mesh\FLEET_PROJECT_ID\PATH_TO_KUBECONFIG_1\PATH_TO_KUBECONFIG_2

Verify multicluster connectivity

This section explains how to deploy the sampleHelloWorld andSleep servicesto your multi-cluster environment to verify that cross-cluster loadbalancing works.

Enable sidecar injection

1. Create the sample namespace in each cluster.

   forCTXin${CTX_1}${CTX_2}dokubectlcreate--context=${CTX}namespacesampledone

2. Enable sidecar injection on created namespaces.

   Recommended: Run the following command to apply the default injection label to the namespace:

   forCTXin${CTX_1}${CTX_2}dokubectllabel--context=${CTX}namespacesample\istio.io/rev-istio-injection=enabled--overwritedone

   We recommend that you use default injection, but revision-based injection is supported:Use the following instructions:

   1. Use the following command to locate the revision label onistiod:

      kubectlgetdeploy-nistio-system-lapp=istiod-o\jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}'

   2. Apply the revision label to the namespace. In the following command,REVISION_LABEL is the value of theistiod revisionlabel that you noted in the previous step.

      forCTXin${CTX_1}${CTX_2}dokubectllabel--context=${CTX}namespacesample\istio-injection-istio.io/rev=REVISION_LABEL--overwritedone

Install the HelloWorld service

• Create the HelloWorld service in both clusters:

  kubectl create --context=${CTX_1} \    -f ${SAMPLES_DIR}/samples/helloworld/helloworld.yaml \    -l service=helloworld -n sample

  kubectl create --context=${CTX_2} \    -f ${SAMPLES_DIR}/samples/helloworld/helloworld.yaml \    -l service=helloworld -n sample

Deploy HelloWorld v1 and v2 to each cluster

1. DeployHelloWorld v1 toCLUSTER_1 andv2 toCLUSTER_2, which helps later to verify cross-cluster load balancing:

   kubectl create --context=${CTX_1} \  -f ${SAMPLES_DIR}/samples/helloworld/helloworld.yaml \  -l version=v1 -n sample

   kubectl create --context=${CTX_2} \  -f ${SAMPLES_DIR}/samples/helloworld/helloworld.yaml \  -l version=v2 -n sample

2. ConfirmHelloWorld v1 andv2 are running using the following commands. Verify that the output is similar to that shown.:

   kubectl get pod --context=${CTX_1} -n sample

   NAME                            READY     STATUS    RESTARTS   AGEhelloworld-v1-86f77cd7bd-cpxhv  2/2       Running   0          40s

   kubectl get pod --context=${CTX_2} -n sample

   NAME                            READY     STATUS    RESTARTS   AGEhelloworld-v2-758dd55874-6x4t8  2/2       Running   0          40s

Deploy the Sleep service

1. Deploy theSleep service to both clusters. This pod generates artificial network traffic for demonstration purposes:

   forCTXin${CTX_1}${CTX_2}dokubectlapply--context=${CTX}\-f${SAMPLES_DIR}/samples/sleep/sleep.yaml-nsampledone

2. Wait for theSleep service to start in each cluster. Verify that the output is similar to that shown:

   kubectl get pod --context=${CTX_1} -n sample -l app=sleep

   NAME                             READY   STATUS    RESTARTS   AGEsleep-754684654f-n6bzf           2/2     Running   0          5s

   kubectl get pod --context=${CTX_2} -n sample -l app=sleep

   NAME                             READY   STATUS    RESTARTS   AGEsleep-754684654f-dzl9j           2/2     Running   0          5s

Verify cross-cluster load balancing

Call theHelloWorld service several times and check the output to verifyalternating replies from v1 and v2:

1. Call theHelloWorld service:

   kubectl exec --context="${CTX_1}" -n sample -c sleep \    "$(kubectl get pod --context="${CTX_1}" -n sample -l \    app=sleep -o jsonpath='{.items[0].metadata.name}')" \    -- /bin/sh -c 'for i in $(seq 1 20); do curl -sS helloworld.sample:5000/hello; done'

   The output is similar to that shown:

   Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv...

2. Call theHelloWorld service again:

   kubectl exec --context="${CTX_2}" -n sample -c sleep \    "$(kubectl get pod --context="${CTX_2}" -n sample -l \    app=sleep -o jsonpath='{.items[0].metadata.name}')" \    -- /bin/sh -c 'for i in $(seq 1 20); do curl -sS helloworld.sample:5000/hello; done'

   The output is similar to that shown:

   Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv...

Congratulations, you've verified your load-balanced, multi-cluster Cloud Service Mesh!

Clean up

When you finish verifying load balancing, remove theHelloWorld andSleepservice from your cluster.

kubectl delete ns sample --context ${CTX_1}kubectl delete ns sample --context ${CTX_2}