This page shows you how to deploy an Ingress that serves an application acrossmultiple GKE clusters. To learn more aboutMulti Cluster Ingress, seeMulti Cluster Ingress.

For a detailed comparison between Multi Cluster Ingress (MCI), Multi-cluster Gateway(MCG), and load balancer with Standalone Network Endpoint Groups (LB and StandaloneNEGs), seeChoose your multi-cluster load balancing API forGKE.

Deployment tutorial

In the following tasks, you will deploy a fictional app namedwhereami and aMultiClusterIngress in two clusters. The Ingress provides a shared virtual IP (VIP)address for the app deployments.

This page builds upon the work done inSetting up Multi Cluster Ingress,where you created and registered two clusters.Confirm you have two clusters that are also registered to afleet:

gcloudcontainerclusterslist

The output is similar to the following:

NAME    LOCATION        MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUSgke-eu  europe-west1-b  1.16.8-gke.9    ***             e2-medium      1.16.8-gke.9     2          RUNNINGgke-us  us-central1-b   1.16.8-gke.9    ***             e2-medium      1.16.6-gke.13 *  2          RUNNING

Creating the Namespace

Because fleets have the property ofnamespace sameness,we recommend that you coordinate Namespace creation and management across clustersso identical Namespaces are owned and managed by the same group. You can createNamespaces per team, per environment, per application, or per applicationcomponent. Namespaces can be as granular as necessary, as long as a Namespacens1 in onecluster has the same meaning and usage asns1 in another cluster.

In this example, you create awhereami Namespace for each application ineach cluster.

1. Create a file namednamespace.yaml that has the following content:

   apiVersion:v1kind:Namespacemetadata:name:whereami

   Note: You can usekubectl config use-context to switch between clusterswhen deploying resources. Usekubectl config get-contexts to see whichones are available. You can usekubectl config rename-contextto rename contexts to more human-friendly names.

2. Switch to the gke-us context:

   kubectlconfiguse-contextgke-us

3. Create the Namespace:

   kubectlapply-fnamespace.yaml

4. Switch to the gke-eu context:

   kubectlconfiguse-contextgke-eu

5. Create the Namespace:

   kubectlapply-fnamespace.yaml

   The output is similar to the following:

   namespace/whereami created

Deploying the app

1. Create a file nameddeploy.yaml that has the following content:

   apiVersion:apps/v1kind:Deploymentmetadata:name:whereami-deploymentnamespace:whereamilabels:app:whereamispec:selector:matchLabels:app:whereamitemplate:metadata:labels:app:whereamispec:containers:-name:frontendimage:us-docker.pkg.dev/google-samples/containers/gke/whereami:v1ports:-containerPort:8080

2. Switch to the gke-us context:

   kubectlconfiguse-contextgke-us

3. Deploy thewhereami app:

   kubectlapply-fdeploy.yaml

4. Switch to the gke-eu context:

   kubectlconfiguse-contextgke-eu

5. Deploy thewhereami app:

   kubectlapply-fdeploy.yaml

6. Verify that thewhereami app has successfully deployed in each cluster:

   kubectlgetdeployment--namespacewhereami

   The output should be similar to the following in both clusters:

   NAME           READY   UP-TO-DATE   AVAILABLE   AGEwhereami-deployment   1/1     1            1           12m

Deploying through the config cluster

Now that the application is deployed acrossgke-us andgke-eu, you willdeploy a load balancer by deployingMultiClusterIngress andMultiClusterService resources in the config cluster. These are themulti-cluster equivalents of Ingress and Service resources.

In thesetup guide,you configured thegke-us cluster as the config cluster. The config cluster isused to deploy and configure Ingress across all clusters.

1. Set the context to the config cluster.

   kubectlconfiguse-contextgke-us

   Note: Only one cluster can be the active config cluster at any time. You candeployMultiClusterIngress andMultiClusterService resources to otherclusters, but they won't be seen or processed by the Multi Cluster Ingresscontroller.

MultiClusterService

1. Create a file namedmcs.yaml that has the following content:

   apiVersion:networking.gke.io/v1kind:MultiClusterServicemetadata:name:whereami-mcsnamespace:whereamispec:template:spec:selector:app:whereamiports:-name:webprotocol:TCPport:8080targetPort:8080

2. Deploy theMultiClusterService resource that matches thewhereami app:

   kubectlapply-fmcs.yaml

3. Verify that thewhereami-mcs resource has successfully deployed in the config cluster:

   kubectlgetmcs-nwhereami

   The output is similar to the following:

   NAME       AGEwhereami-mcs   9m26s

   ThisMultiClusterService creates a derived headless Service in every cluster that matches Pods withapp: whereami. You can see that one exists in thegke-us clusterkubectl get service -n whereami.

   The output is similar to the following:

   NAME                                TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGEmci-whereami-mcs-svc-lgq966x5mxwwvvum   ClusterIP   None          <none>        8080/TCP         4m59s

A similar headless Service will also exist ingke-eu. These local Services areused to dynamically select Pod endpoints to program the global Ingress loadbalancer with backends.

MultiClusterIngress

1. Create a file namedmci.yaml that has the following content:

   apiVersion:networking.gke.io/v1kind:MultiClusterIngressmetadata:name:whereami-ingressnamespace:whereamispec:template:spec:backend:serviceName:whereami-mcsservicePort:8080

   Note that this configuration routes all traffic to theMultiClusterServicenamedwhereami-mcs that exists in thewhereami namespace.

2. Deploy theMultiClusterIngress resource that referenceswhereami-mcs as a backend:

   kubectlapply-fmci.yaml

   The output is similar to the following:

   multiclusteringress.networking.gke.io/whereami-ingress created

   Note thatMultiClusterIngress has the same schema as the Kubernetes Ingress.The Ingress resource semantics are also the same with the exception of thebackend.serviceName field.

Thebackend.serviceName field in aMultiClusterIngress references aMultiClusterService in the fleet API rather than a Service in a Kubernetescluster. This means that any of the settings for Ingress, such as TLStermination, settings can be configured in the same way.

Validating a successful deployment status

Google Cloud Load Balancer deployment might take several minutes to deploy for newload balancers. Updating existing load balancers completes faster because newresources don't need to be deployed. TheMultiClusterIngress resource details the underlyingCompute Engine resources that have been created on behalf of theMultiClusterIngress.

1. Verify that deployment has succeeded:

   kubectldescribemciwhereami-ingress-nwhereami

   The output is similar to the following:

   Name:         whereami-ingressNamespace:    whereamiLabels:       <none>Annotations:  kubectl.kubernetes.io/last-applied-configuration:                {"apiVersion":"networking.gke.io/v1","kind":"MultiClusterIngress","metadata":{"annotations":{},"name":"whereami-ingress","namespace":"whe...API Version:  networking.gke.io/v1Kind:         MultiClusterIngressMetadata:  Creation Timestamp:  2020-04-10T23:35:10Z  Finalizers:    mci.finalizer.networking.gke.io  Generation:        2  Resource Version:  26458887  Self Link:         /apis/networking.gke.io/v1/namespaces/whereami/multiclusteringresses/whereami-ingress  UID:               62bec0a4-8a08-4cd8-86b2-d60bc2bda63dSpec:  Template:    Spec:      Backend:        Service Name:  whereami-mcs        Service Port:  8080Status:  Cloud Resources:    Backend Services:      mci-8se3df-8080-whereami-whereami-mcs    Firewalls:      mci-8se3df-default-l7    Forwarding Rules:      mci-8se3df-fw-whereami-whereami-ingress    Health Checks:      mci-8se3df-8080-whereami-whereami-mcs    Network Endpoint Groups:      zones/europe-west1-b/networkEndpointGroups/k8s1-e4adffe6-whereami-mci-whereami-mcs-svc-lgq966x5m-808-88670678      zones/us-central1-b/networkEndpointGroups/k8s1-a6b112b6-whereami-mci-whereami-mcs-svc-lgq966x5m-808-609ab6c6    Target Proxies:      mci-8se3df-whereami-whereami-ingress    URL Map:  mci-8se3df-whereami-whereami-ingress  VIP:        34.98.102.37Events:  Type    Reason  Age                    From                              Message  ----    ------  ----                   ----                              -------  Normal  ADD     3m35s                  multi-cluster-ingress-controller  whereami/whereami-ingress  Normal  UPDATE  3m10s (x2 over 3m34s)  multi-cluster-ingress-controller  whereami/whereami-ingress

   There are several fields that indicate the status of this Ingress deployment:

   • Events is the first place to look. If an error has occurred it will belisted here.

   • Cloud Resourcelists the Compute Engine resources like forwardingrules, backend services, and firewall rules that have been created by theMulti Cluster Ingress controller. If these are not listed it means that theyhave not been created yet. You can inspect individual Compute Engineresources with the Console orgcloud command to get its status.

   • VIP lists an IP address when one has been allocated. Note that the loadbalancer may not yet be processing traffic even though the VIP exists. If youdon't see a VIP after a couple minutes, or if the load balancer is not servinga 200 response within 10 minutes, seeTroubleshooting and operations.

   If the output events areNormal, then theMultiClusterIngress deployment is likelysuccessful, but the only way to determine that the full traffic path isfunctional is to test it.

2. Validate that the application is serving on the VIP with the/ping endpoint:

   curlINGRESS_VIP/ping

   ReplaceINGRESS_VIP with the virtual IP (VIP) address.

   The output is similar to the following:

   {"cluster_name": "gke-us","host_header": "34.120.175.141","pod_name": "whereami-deployment-954cbf78-mtlpf","pod_name_emoji": "😎","project_id": "my-project","timestamp": "2021-11-29T17:01:59","zone": "us-central1-b"}

   The output should indicate the region and backend of the application.

3. You can also go to thehttp://INGRESS_VIPURL in your browser to see a graphical version of the application that showsthe region that it's being served from.

   The cluster that the traffic is forwarded to depends on your location. TheGCLB is designed to forward client traffic to the closest available backendwith capacity.

Resource specs

MultiClusterService spec

TheMultiClusterService definition consists of two pieces:

1. Atemplate section that defines the Service to be created in the Kubernetesclusters. Note that while thetemplate section contains fields supportedin a typical Service, there are only two fields that aresupported in aMultiClusterService:selector andports. The other fieldsare ignored.

2. An optionalclusters section that defines which clusters receive trafficand the load balancing properties for each cluster. If noclusters sectionis specified or if no clusters are listed, all clusters are used by default.

The following manifest describes a standardMultiClusterService:

apiVersion:networking.gke.io/v1kind:MultiClusterServicemetadata:name:NAMEnamespace:NAMESPACEspec:template:spec:selector:app:POD_LABELports:-name:webprotocol:TCPport:PORTtargetPort:TARGET_PORT

Replace the following:

• NAME: the name of theMultiClusterService. This name is referenced by theserviceName field in theMultiClusterIngress resources.
• NAMESPACE: the Kubernetes Namespace that theMultiClusterService is deployed in.It must match be in the same Namespace as theMultiClusterIngress and the Pods across allclusters in the fleet.
• POD_LABEL: the label that determines which pods are selected asbackends for thisMultiClusterService across all clusters in the fleet.
• PORT: must match with the port referenced by theMultiClusterIngress that referencesthisMultiClusterService.
• TARGET_PORT: the port that is used to send traffic to the Podfrom the GCLB. A NEG is created in each cluster with this port as its servingport.

MultiClusterIngress spec

The followingmci.yaml describes the load balancer frontend:

apiVersion:networking.gke.io/v1kind:MultiClusterIngressmetadata:name:NAMEnamespace:NAMESPACEspec:template:spec:backend:serviceName:DEFAULT_SERVICEservicePort:PORTrules:-host:HOST_HEADERhttp:paths:-path:PATHbackend:serviceName:SERVICEservicePort:PORT

Replace the following:

• NAME: the name of theMultiClusterIngress resource.
• NAMESPACE: the Kubernetes Namespace that theMultiClusterIngress is deployed in.It must be in the same Namespace as theMultiClusterService and the Pods across all clustersin the fleet.
• DEFAULT_SERVICE: acts as the default backend for all traffic thatdoes not match any host or path rules. This is a required field and a defaultbackend must be specified in theMultiClusterIngress even if there are other host or pathmatches configured.
• PORT: any valid port number. This must match with theportfield of theMultiClusterService resources.
• HOST_HEADER: matches traffic by the HTTP host header field. Thehost field is optional.
• PATH: matches traffic by the path of the HTTP URL. Thepath fieldis optional.
• SERVICE: the name of aMultiClusterService that is deployed in the sameNamespace and config cluster as thisMultiClusterIngress.

Multi Cluster Ingress features

This section shows you how to configure additional Multi Cluster Ingress features.

Cluster selection

By default, Services derived from Multi Cluster Ingress are scheduled on everymember cluster. However, you may want to apply ingress rules tospecific clusters. Some use-cases include:

• Applying Multi Cluster Ingress to all clusters but the config cluster forisolation of the config cluster.
• Migrating workloads between clusters in a blue-green fashion.
• Routing to application backends that only exist in a subset of clusters.
• Using a single L7 VIP for host or path routing to backends that live on different clusters.

Cluster selection lets you select clusters by region or name in theMultiClusterService object. This controls which clusters yourMultiClusterIngress is pointing to and where the derived Services are scheduled.Clusters within the same fleet and region shouldn't have the same name so thatclusters can be referenced uniquely.

1. Openmcs.yaml

   apiVersion:networking.gke.io/v1kind:MultiClusterServicemetadata:name:whereami-mcsnamespace:whereamispec:template:spec:selector:app:whereamiports:-name:webprotocol:TCPport:8080targetPort:8080

   This specification creates Derived Services in all clusters, thedefault behavior.

2. Append the following lines in the clusters section:

   apiVersion:networking.gke.io/v1kind:MultiClusterServicemetadata:name:whereami-mcsnamespace:whereamispec:template:spec:selector:app:whereamiports:-name:webprotocol:TCPport:8080targetPort:8080clusters:-link:"us-central1-b/gke-us"-link:"europe-west1-b/gke-eu"

   This example creates Derived Service resources only in gke-us and gke-euclusters. You must select clusters to selectively apply ingress rules. If the"clusters" section of theMultiClusterService is not specified or if noclusters are listed, it is interpreted as the default "all" clusters.

HTTPS support

The KubernetesSecret supports HTTPS. Before enabling HTTPS support, you must create a static IPaddress. This static IP allows HTTP and HTTPS to share the same IP address. Formore information, seeCreating a static IP.

Once you have created a static IP address, you can create a Secret.

1. Create a Secret:

   kubectl-nwhereamicreatesecrettlsSECRET_NAME--keyPATH_TO_KEYFILE--certPATH_TO_CERTFILE

   Replace the following:

   • SECRET_NAME with the name of your Secret.
   • PATH_TO_KEYFILE with the path to the TLS key file.
   • PATH_TO_CERTFILE with the path to the TLS certificate file.

2. Update themci.yaml file with the Secret name:

   apiVersion:networking.gke.io/v1kind:MultiClusterIngressmetadata:name:whereami-ingressnamespace:whereamiannotations:networking.gke.io/static-ip:STATIC_IP_ADDRESSspec:template:spec:backend:serviceName:whereami-mcsservicePort:8080tls:-secretName:SECRET_NAME

   Replace theSECRET_NAME with the name of your Secret.TheSTATIC_IP_ADDRESS is the IP address or thecomplete URL of the address you allocated in theCreating a staticIP section.

3. Redeploy theMultiClusterIngress resource:

   kubectlapply-fmci.yaml

   The output is similar to the following:

   multiclusteringress.networking.gke.io/whereami-ingress configured

BackendConfig support

The following BackendConfig CRD lets you customize settings on theCompute Engine BackendService resource:

apiVersion:cloud.google.com/v1kind:BackendConfigmetadata:name:whereami-health-check-cfgnamespace:whereamispec:healthCheck:checkIntervalSec:[int]timeoutSec:[int]healthyThreshold:[int]unhealthyThreshold:[int]type:[HTTP | HTTPS | HTTP2 | TCP]port:[int]requestPath:[string]timeoutSec:[int]connectionDraining:drainingTimeoutSec:[int]sessionAffinity:affinityType:[CLIENT_IP | CLIENT_IP_PORT_PROTO | CLIENT_IP_PROTO | GENERATED_COOKIE | HEADER_FIELD | HTTP_COOKIE | NONE]affinityCookieTtlSec:[int]cdn:enabled:[bool]cachePolicy:includeHost:[bool]includeQueryString:[bool]includeProtocol:[bool]queryStringBlacklist:[string list]queryStringWhitelist:[string list]securityPolicy:name:ca-how-to-security-policylogging:enable:[bool]sampleRate:[float]iap:enabled:[bool]oauthclientCredentials:secretName:[string]

To use BackendConfig, attach it on yourMultiClusterService resource using an annotation:

apiVersion:networking.gke.io/v1kind:MultiClusterServicemetadata:name:whereami-mcsnamespace:whereamiannotations:cloud.google.com/backend-config:'{"ports":{"8080":"whereami-health-check-cfg"}}'spec:template:spec:selector:app:whereamiports:-name:webprotocol:TCPport:8080targetPort:8080

For more information about BackendConfig semantics, seeAssociating a service port with a BackendConfig.

gRPC support

Configuring gRPC applications on Multi Cluster Ingress requires very specific setup.Here are some tips to make sureyour load balancer is configured properly:

1. Make sure that the traffic from the load balancer to your application isHTTP/2. Useapplication protocols to configure this.
2. Make sure that your application is properly configured for SSL since this is arequirement of HTTP/2. Note that using self-signed certs is acceptable.
3. You must turn off mTLS on your application because mTLS isnot supported for L7external load balancers.

Resource lifecycle

Configuration changes

MultiClusterIngress andMultiClusterService resources behave as standardKubernetes objects, so changes to the objects are asynchronously reflected inthe system. Any changes that result in an invalid configuration cause associatedGoogle Cloud objects to remain unchanged and raise an error in the object eventstream. Errors associated with the configuration will be reported as events.

Managing Kubernetes resources

Deleting the Ingress object tears down the HTTP(S) load balancer sotraffic is no longer forwarded to any definedMultiClusterService.

Deleting theMultiClusterService removes the associated derived services ineach of the clusters.

Managing clusters

The set of clusters targeted by the load balancer can be changed by adding orremoving clusters from the fleet.

For example, to remove thegke-eu cluster as a backend for an ingress,run:

gcloudcontainerfleetmembershipsunregisterCLUSTER_NAME\--gke-uri=URI

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• URI: the URI of the GKE cluster.

To add a cluster in Europe, run:

gcloudcontainerfleetmembershipsregistereurope-cluster\--context=europe-cluster--enable-workload-identity

You can find out more about cluster registration options inRegister a GKE cluster.

Note that registering or unregistering a cluster changes its status as a backendfor all Ingresses. Unregistering thegke-eu clusterremoves it as an available backend for all Ingresses you create. Thereverse is true for registering a new cluster.

gcloudcontainerfleetingressdisable

Feature has associated resources that should be cleaned up before deletion.

gcloudcontainerfleetingressdisable--force

1. Create anSSL policy that will reject requests using a version lower than TLS v1.2.

   gcloudcomputessl-policiescreatetls-12-policy\--profileMODERN\--min-tls-version1.2\--project=PROJECT_ID

   ReplacePROJECT_ID with the project ID where yourGKE clusters are running.

2. View your policy to ensure it has been created.

   gcloudcomputessl-policieslist--project=PROJECT_ID

   The output is similar to the following:

   NAME           PROFILE  MIN_TLS_VERSIONtls-12-policy  MODERN   TLS_1_2

3. Create a certificate forfoo.example.com as inthe example. Once you have thekey.pemandcert.pem, store these credentials as a Secret that will be referenced by the MultiClusterIngress resource.

   kubectl-nwhereamicreatesecrettlsSECRET_NAME--keykey.pem--certcert.pem

4. Save the following FrontendConfig resource asfrontendconfig.yaml. SeeConfiguring FrontendConfig resourcesfor more information on the supported fields within a FrontendConfig.

   apiVersion:networking.gke.io/v1beta1kind:FrontendConfigmetadata:name:frontend-redirect-tls-policynamespace:whereamispec:sslPolicy:tls-12-policyredirectToHttps:enabled:true

   This FrontendConfig will enable HTTPS redirects and an SSL policythat enforces a minimum TLS version of 1.2.

5. Deployfrontendconfig.yaml into your config cluster.

   kubectlapply-ffrontendconfig.yaml--contextMCI_CONFIG_CLUSTER

   Replace theMCI_CONFIG_CLUSTER with the name of yourconfig cluster.

6. Save the following MultiClusterIngress asmci-frontendconfig.yaml.

   apiVersion:networking.gke.io/v1kind:MultiClusterIngressmetadata:name:foo-ingressnamespace:whereamiannotations:networking.gke.io/frontend-config:frontend-redirect-tls-policynetworking.gke.io/static-ip:STATIC_IP_ADDRESSspec:template:spec:backend:serviceName:default-backendservicePort:8080rules:-host:foo.example.comhttp:paths:-backend:serviceName:whereami-mcsservicePort:8080tls:-secretName:SECRET_NAME

   • ReplaceSTATIC_IP_ADDRESS with a static global IP address that you have already provisioned.
   • ReplaceSECRET_NAME with the Secret where yourfoo.example.com certificate is stored.

   There are two requirements when enabling HTTPS redirects:

   • TLS must be enabled, either through thespec.tls field or through thepre-shared certificate annotationnetworking.gke.io/pre-shared-certs. TheMultiClusterIngress won't deploy if HTTPS redirects is enabled but HTTPSis not.
   • A static IP must be referenced through thenetworking.gke.io/static-ipannotation. Static IPs are required when enabling HTTPS on a MultiClusterIngress.
   Note: Deployment of the application or MultiClusterService is not shown inthis example, but they must also be created for this to fully work.

7. Deploy the MultiClusterIngress to your config cluster.

   kubectlapply-fmci-frontendconfig.yaml--contextMCI_CONFIG_CLUSTER

8. Wait a minute or two and inspectfoo-ingress.

   kubectldescribemcifoo-ingress--contextMCI_CONFIG_CLUSTER

   A successful output resembles the following:

   • TheCloud Resources status is populated with resource names
   • TheVIP field is populated with the load balancer IP address

   Name:         foobar-ingressNamespace:    whereami...Status:  Cloud Resources:    Backend Services:      mci-otn9zt-8080-whereami-bar      mci-otn9zt-8080-whereami-default-backend      mci-otn9zt-8080-whereami-foo    Firewalls:      mci-otn9zt-default-l7    Forwarding Rules:      mci-otn9zt-fw-whereami-foobar-ingress      mci-otn9zt-fws-whereami-foobar-ingress    Health Checks:      mci-otn9zt-8080-whereami-bar      mci-otn9zt-8080-whereami-default-backend      mci-otn9zt-8080-whereami-foo    Network Endpoint Groups:      zones/europe-west1-b/networkEndpointGroups/k8s1-1869d397-multi-cluste-mci-default-backend-svc--80-9e362e3d      zones/europe-west1-b/networkEndpointGroups/k8s1-1869d397-multi-cluster--mci-bar-svc-067a3lzs8-808-89846515      zones/europe-west1-b/networkEndpointGroups/k8s1-1869d397-multi-cluster--mci-foo-svc-820zw3izx-808-8bbcb1de      zones/us-central1-b/networkEndpointGroups/k8s1-a63e24a6-multi-cluste-mci-default-backend-svc--80-a528cc75      zones/us-central1-b/networkEndpointGroups/k8s1-a63e24a6-multi-cluster--mci-bar-svc-067a3lzs8-808-36281739      zones/us-central1-b/networkEndpointGroups/k8s1-a63e24a6-multi-cluster--mci-foo-svc-820zw3izx-808-ac733579    Target Proxies:      mci-otn9zt-whereami-foobar-ingress      mci-otn9zt-whereami-foobar-ingress    URL Map:  mci-otn9zt-rm-whereami-foobar-ingress  VIP:        34.149.29.76Events:  Type     Reason  Age                From                              Message  ----     ------  ----               ----                              -------  Normal   UPDATE  38m (x5 over 62m)  multi-cluster-ingress-controller  whereami/foobar-ingress

9. Verify that HTTPS redirects function correctly by sending an HTTP requestthroughcurl.

   curlVIP

   ReplaceVIP with the MultiClusterIngress IP address.

   The output should show that the request was redirected to the HTTPS portwhich indicates that redirects are functioning correctly.

10. Verify that the TLS policy functions correctly by sending an HTTPS requestusing TLS version 1.1. Because DNS is not configured for this domain, use the--resolve option to tellcurl to resolve the IP address directly.

    curlhttps://foo.example.com--resolvefoo.example.com:443:VIP--cacertCERT_FILE-v

    This step requires the certificate PEM file used to secure the MultiClusterIngress.A successful output will look similar to the following:

    ...* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305* ALPN, server accepted to use h2* Server certificate:*  subject: O=example; CN=foo.example.com*  start date: Sep  1 10:32:03 2021 GMT*  expire date: Aug 27 10:32:03 2022 GMT*  common name: foo.example.com (matched)*  issuer: O=example; CN=foo.example.com*  SSL certificate verify ok.* Using HTTP2, server supports multi-use* Connection state changed (HTTP/2 confirmed)* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0* Using Stream ID: 1 (easy handle 0x7fa10f00e400)> GET / HTTP/2> Host: foo.example.com> User-Agent: curl/7.64.1> Accept: */*>* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!< HTTP/2 200< content-type: application/json< content-length: 308< access-control-allow-origin: *< server: Werkzeug/1.0.1 Python/3.8.6< date: Wed, 01 Sep 2021 11:39:06 GMT< via: 1.1 google< alt-svc: clear<{"cluster_name":"gke-us","host_header":"foo.example.com","metadata":"foo","node_name":"gke-gke-us-default-pool-22cb07b1-r5r0.c.mark-church-project.internal","pod_name":"foo-75ccd9c96d-dkg8t","pod_name_emoji":"👞","project_id":"mark-church-project","timestamp":"2021-09-01T11:39:06","zone":"us-central1-b"}* Connection #0 to host foo.example.com left intact* Closing connection 0

    The response code is 200 and TLSv1.2 is being used which indicates thateverything is functioning properly.

    Next you can verify that the SSL policy enforces the correct TLS version byattempting to connect with TLS 1.1. Your SSL policy must be configuredfor a minimum version of 1.2 for this step to work.

11. Send the same request from the previous step, but enforce a TLS version of 1.1.

    curlhttps://foo.example.com--resolvefoo.example.com:443:VIP-v\--cacertCERT_FILE\--tls-max1.1

    A successful output will look similar to the following:

    * Added foo.example.com:443:34.149.29.76 to DNS cache* Hostname foo.example.com was found in DNS cache*   Trying 34.149.29.76...* TCP_NODELAY set* Connected to foo.example.com (34.149.29.76) port 443 (#0)* ALPN, offering h2* ALPN, offering http/1.1* successfully set certificate verify locations:*   CAfile: cert.pem  CApath: none* TLSv1.1 (OUT), TLS handshake, Client hello (1):* TLSv1.1 (IN), TLS alert, protocol version (582):* error:1400442E:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert protocol version* Closing connection 0curl: (35) error:1400442E:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert protocol version

    The failure to complete the TLS handshake indicates that the SSL policy hasblocked TLS 1.1 successfully.

Creating a static IP

1. Allocate a static IP:

   gcloudcomputeaddressescreateADDRESS_NAME--global

   ReplaceADDRESS_NAME with the name of the static IP to allocate.

   The output contains the complete URL of the address you created, similar to the following:

   Created [https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/addresses/ADDRESS_NAME].

2. View the IP address you just created:

   gcloudcomputeaddresseslist

   The output is similar to the following:

   NAME          ADDRESS/RANGE  TYPE      STATUSADDRESS_NAMESTATIC_IP_ADDRESS  EXTERNAL  RESERVED

   This output includes:

   • TheADDRESS_NAME you defined.
   • TheSTATIC_IP_ADDRESS allocated.

3. Update themci.yaml file with the static IP:

   apiVersion:networking.gke.io/v1kind:MultiClusterIngressmetadata:name:whereami-ingressnamespace:whereamiannotations:networking.gke.io/static-ip:STATIC_IP_ADDRESSspec:template:spec:backend:serviceName:whereami-mcsservicePort:8080

   Replace theSTATIC_IP_ADDRESS with either:

   • The allocated IP address, similar to:34.102.201.47
   • The complete URL of the address you created, similar to:"https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/addresses/ADDRESS_NAME"

   TheSTATIC_IP_ADDRESS isnot the resource name (ADDRESS_NAME).

4. Redeploy theMultiClusterIngress resource:

   kubectlapply-fmci.yaml

   The output is similar to the following:

   multiclusteringress.networking.gke.io/whereami-ingress configured

5. Follow the steps inValidating a successful deployment statusto verify that the deployment is serving on theSTATIC_IP_ADDRESS.

Pre-shared certificates

Pre-shared certificatesare certificates uploaded to Google Cloud that can be used by the loadbalancer for TLS termination instead of certificates stored in KubernetesSecrets. These certificates are uploaded out of band from GKEto Google Cloud and referenced by aMultiClusterIngress resource.Multiple certificates, either through pre-shared certs or Kubernetes secrets,are also supported.

Using the certificates in Multi Cluster Ingress requires thenetworking.gke.io/pre-shared-certs annotation and the names of the certs. Whenmultiple certificates are specified for a givenMultiClusterIngress, apredetermined order governs which cert is presented to the client.

You can list the available SSL certificates by running:

gcloudcomputessl-certificateslist

The following example describes client traffic to one of the specified hosts thatmatches the Common Name of the pre-shared certs so the respective certificatethat matches the domain name will be presented.

kind:MultiClusterIngressmetadata:name:shopping-servicenamespace:whereamiannotations:networking.gke.io/pre-shared-certs:"domain1-cert,domain2-cert"spec:template:spec:rules:-host:my-domain1.gcp.comhttp:paths:-backend:serviceName:domain1-svcservicePort:443-host:my-domain2.gcp.comhttp:paths:-backend:serviceName:domain2-svcservicePort:443

Google-managed Certificates

Google-managed Certificatesare supported onMultiClusterIngress resources through thenetworking.gke.io/pre-shared-certsannotation. Multi Cluster Ingress supports the attachment of Google-managedcertificates to aMultiClusterIngress resource, however unlike single-clusterIngress, thedeclarative generation of a KubernetesManagedCertificate resourceis not supported onMultiClusterIngress resources. The original creation ofthe Google-managed certificate must be done directly through thecompute ssl-certificates create API before you can attach it to aMultiClusterIngress. That can be done following these steps:

1. Create a Google-managed Certificate asin step 1 here.Don't move to step 2 as Multi Cluster Ingress will attach this certificate for you.

   gcloudcomputessl-certificatescreatemy-google-managed-cert\--domains=my-domain.gcp.com\--global

2. Reference the name of the certificate in yourMultiClusterIngress using thenetworking.gke.io/pre-shared-certs annotation.

   kind:MultiClusterIngressmetadata:name:shopping-servicenamespace:whereamiannotations:networking.gke.io/pre-shared-certs:"my-google-managed-cert"spec:template:spec:rules:-host:my-domain.gcp.comhttp:paths:-backend:serviceName:my-domain-svcservicePort:8080

The preceding manifest attaches the certificate to yourMultiClusterIngressso that it can terminate traffic for your backend GKE clusters.Google Cloud willautomatically renew your certificateprior to certificate expiry. Renewals occur transparently and does not require anyupdates to Multi Cluster Ingress.

Application protocols

The connection from the load balancer proxy to your application uses HTTP bydefault. Usingnetworking.gke.io/app-protocols annotation, you can configurethe load balancer to use HTTPS or HTTP/2 when it forwards requests to yourapplication. In theannotation field of the following example,http2refers to theMultiClusterService port name andHTTP2 refers to theprotocol that the load balancer uses.

kind:MultiClusterServicemetadata:name:shopping-servicenamespace:whereamiannotations:networking.gke.io/app-protocols:'{"http2":"HTTP2"}'spec:template:spec:ports:-port:443name:http2

BackendConfig

Refer to the sectionabove on how to configure theannotation.

What's next

• Read the GKE network overview.
• Learn more aboutsetting up HTTP Load Balancing with Ingress.
• ImplementMulti Cluster Ingress with end to end HTTPS.