Cloud Service Mesh with hybrid connectivity network endpoint groups

Cloud Service Mesh supports environments that extend beyond Google Cloud,including on-premises data centers and other public clouds that you can usehybrid connectivity to reach.

You configure Cloud Service Mesh so that your service mesh can send traffic toendpoints that are outside of Google Cloud. These endpoints includethe following:

• On-premises load balancers.
• Server applications on a virtual machine (VM) instance in another cloud.
• Any other destination that you can reach with hybrid connectivity and thatcan be reached with an IP address and a port.

You add each endpoint's IP address and port to a hybrid connectivity networkendpoint group (NEG). Hybrid connectivity NEGs are of typeNON_GCP_PRIVATE_IP_PORT.

Cloud Service Mesh's support for on-premises and multi-cloud serviceslets you do the following:

• Route traffic globally, including to the endpoints of on-premises andmulti-cloud services.
• Bring the benefits of Cloud Service Mesh and service mesh—includingcapabilities such as service discovery and advanced traffic management—toservices running on your existing infrastructure outside of Google Cloud.
• Combine Cloud Service Mesh capabilities with Cloud Load Balancing tobring Google Cloud networking services to multi-environments.

Hybrid connectivity NEGs (NON_GCP_PRIVATE_IP_PORT NEGs) are not supported withproxyless gRPC clients.

Use cases

Cloud Service Mesh can configure networking between VM-based and container-basedservices in multiple environments, including:

• Google Cloud
• On-premises data centers
• Other public clouds

Route mesh traffic to an on-premises location or another cloud

The simplest use case for this feature is traffic routing. Yourapplication is running a Cloud Service Mesh Envoy proxy . Cloud Service Mesh tellsthe client about your services and each service's endpoints.

In the preceding diagram, when your application sends a request to theon-premservice, the Cloud Service Mesh client inspects the outbound request and updatesits destination. The destination gets set to an endpoint associated with theon-prem service (in this case,10.2.0.1). The request then travels overCloud VPN orCloud Interconnectto its intended destination.

If you need to add more endpoints, you update Cloud Service Mesh to add them toyour service. You don't need to change your application code.

Migrate an existing on-premises service to Google Cloud

Sending traffic to a non-Google Cloud endpoint lets you route traffic toother environments. You can combine this capability with advanced trafficmanagement to migrate services between environments.

The preceding diagram extends the previous pattern. Instead of configuringCloud Service Mesh to send all traffic to theon-prem service, youconfigure Cloud Service Mesh to use weight-based traffic splitting to splittraffic across two services.

Traffic splitting lets you start by sending 0% of traffic to thecloudservice and 100% to theon-prem service. You can then gradually increase theproportion of traffic sent to thecloud service. Eventually, you send 100% oftraffic to thecloud service, and you can retire theon-prem service.

Google Cloud network edge services for on-premises and multi-cloud deployments

Finally, you can combine this functionality with Google Cloud's existingnetworking solutions. Google Cloud offers a wide range of networkservices, such as global external load balancing with Google Cloud Armor fordistributed denial-of-service (DDoS) protection, that you can use withCloud Service Mesh to bring new capabilities to your on-premises or multi-cloudservices. Best of all, you don't need to expose these on-premises or multi-cloudservices to the public internet.

In the preceding diagram, traffic from clients on the public internet entersGoogle Cloud's network from a Google Cloud load balancer, such asour global external Application Load Balancer. When traffic reaches the load balancer, you canapply network edge services such as Cloud Armor DDoS protection orIdentity-Aware Proxy (IAP) user authentication. For more information, seeNetwork edge services for multi-environment deployments.

After you apply these services, the traffic makes a brief stop inGoogle Cloud, where an application or standalone proxy (configured byCloud Service Mesh) forwards the traffic across Cloud VPN orCloud Interconnect to your on-premises service.

Google Cloud resources and architecture

This section provides background information about the Google Cloudresources that you can use to provide a Cloud Service Mesh-managed service meshfor on-premises and multi-cloud environments.

The following diagram depicts the Google Cloud resources that enableon-premises and multi-cloud services support for Cloud Service Mesh. The keyresource is the NEG and its network endpoints. The other resources arethe resources that you configure as part of a standard Cloud Service Mesh setup.For simplicity, the diagram does not show options such as multiple globalbackend services.

When you configure Cloud Service Mesh, you use the global backend services APIresource to create services. A service is a logical construct thatcombines the following:

1. Policies to apply when a client tries to send traffic to the service.
2. One or more backends or endpoints that handle the traffic that is destinedfor the service.

On-premises and multi-cloud services are like any other service thatCloud Service Mesh configures. The key difference is that you use ahybridconnectivity NEG to configure the endpoints of these services. These NEGshave thenetwork endpoint type set toNON_GCP_PRIVATE_IP_PORT. Theendpoints that you add to hybrid connectivity NEGs must be validIP:portcombinations that your clients can reach—forexample, through hybrid connectivity such as Cloud VPN orCloud Interconnect.

Each NEG has a network endpoint type and can only contain networkendpoints of the same type. This type determines the following:

• The destination to which your services can send traffic.
• Health checking behavior.

When you create your NEG, configure it as follows so that you can send trafficto an on-premises or multi-cloud destination.

• Set the network endpoint type toNON_GCP_PRIVATE_IP_PORT. Thisrepresents a reachable IP address. If this IP address is on-premises or atanother cloud provider, it must be reachable from Google Cloud by usinghybrid connectivity, such as the connectivity provided by Cloud VPNor Cloud Interconnect.
• Specify a Google Cloudzonethat minimizes the geographic distance between Google Cloud and youron-premises or multi-cloud environment. For example, if you are hosting aservice in an on-premises environment in Frankfurt, Germany, you canspecify theeurope-west3-a Google Cloud zone when you create the NEG.

Health checking behavior for network endpoints of this type differs fromhealth checking behavior for other types of network endpoints. While othernetwork endpoint types use Google Cloud's centralized health checkingsystem,NON_GCP_PRIVATE_IP_PORT network endpoints use Envoy's distributedhealth checking mechanism. For more details, see theLimitations and other considerations section.

Connectivity and networking considerations

Your Cloud Service Mesh clients, such as Envoy proxies, must be able to connectto Cloud Service Mesh attrafficdirector.googleapis.com:443. If you loseconnectivity to the Cloud Service Mesh control plane, the following happens:

• Existing Cloud Service Mesh clients cannot receive configurationupdates from Cloud Service Mesh. They continue to operate based on theircurrent configuration.
• New Cloud Service Mesh clients cannot connect to Cloud Service Mesh. Theycannot use the service mesh until connectivity is re-established.

If you want to send traffic between Google Cloud and on-premises ormulti-cloud environments, the environments must be connected through hybridconnectivity. We recommend a high availability connection enabled byCloud VPN or Cloud Interconnect.

On-premises, other cloud, and Google Cloud subnet IP addresses andIP address ranges must not overlap.

Limitations and other considerations

The following are limitations of using hybrid connectivity NEGs.

SettingproxyBind

You can only set the value ofproxyBind when you create atargetHttpProxy.You can't update an existingtargetHttpProxy.

Connectivity and disruption to connectivity

For details about connectivity requirements and limitations, see theConnectivity and networking considerations section.

Mixed backend types

A backend service can have VM or NEG backends. If a backend service has NEGbackends, all NEGs must contain the same network endpoint type. You cannot havea backend service with multiple NEGs, each with different endpoint types.

A URL map can have host rules that resolve to different backendservices. You might have a backend service with only hybrid connectivity NEGs(with on-premises endpoints) and a backend service with standalone NEGs(with GKE endpoints). The URL map can contain rules, for example,weight-based traffic splitting, that split traffic across each of these backendservices.

Using a NEG with endpoints of typeNON_GCP_PRIVATE_IP_PORT with Google Cloud backends

It is possible to create a backend service with a hybrid connectivity NEG thatpoints to backends in Google Cloud. However, we do not recommend thispattern because hybrid connectivity NEGs don't benefit from centralized healthchecking. For an explanation of centralized health checking and distributedhealth checking, see theHealth checking section.

Endpoint registration

If you want to add an endpoint to a NEG, you must update the NEG. This caneither be done manually, or it can be automated by using the Google CloudNEG REST APIs or the Google Cloud CLI.

When a new instance of a service starts, you can use the Google Cloud APIsto register the instance with the NEG that you configured. Whenusing Compute Engine managed instance groups (MIGs) or GKE(in Google Cloud), the MIG or NEG controller automatically handlesendpoint registration, respectively.

Health checking

When you use hybrid connectivity NEGs, health checking behavior differs from thestandard centralized health checking behavior in the following ways:

• For network endpoints of typeNON_GCP_PRIVATE_IP_PORT, Cloud Service Meshconfigures its clients to use the data plane to handle health checking.To avoid sending requests to unhealthy backends, Envoy instancesperform their own health checks and use their own mechanisms.
• Because your data plane handles health checks, you cannot use theGoogle Cloud console, the API, or the Google Cloud CLI to retrieve healthcheck status.

In practice, usingNON_GCP_PRIVATE_IP_PORT means the following:

• Because Cloud Service Mesh clients each handle health checking in adistributed fashion, you might see an increase in network traffic because ofhealth checking. The increase depends on the number of Cloud Service Meshclients and the number of endpoints that each client needs to healthcheck. For example:
  • When you add another endpoint to a hybrid connectivity NEG, existingCloud Service Mesh clients might begin to health check the endpointsin hybrid connectivity NEGs.
  • When you add another instance to your service mesh (for example, a VMinstance that runs your application code as well as aCloud Service Mesh client), the new instance might begin to healthcheck the endpoints in hybrid connectivity NEGs.
  • Network traffic because of health checks increases at a quadratic(O(n^2)) rate.

VPC network

A service mesh is uniquely identified by its Virtual Private Cloud (VPC) networkname. Cloud Service Mesh clients receive configuration from Cloud Service Meshbased on the VPC network specified in the bootstrapconfiguration. Therefore, even if your mesh is entirely outside of aGoogle Cloud data center, you must supply a valid VPCnetwork name in your bootstrap configuration.

Service account

Within Google Cloud, the default Envoy bootstrap is configured to readservice account information from either or both of the Compute Engine andGKE deployment environments. When running outside ofGoogle Cloud, you must explicitly specify a service account, network name,and project number in your Envoy bootstrap. This service account must havesufficient permissions to connect with the Cloud Service Mesh API.

What's next

• To configure Cloud Service Mesh for on-premises and multi-cloud deployments, seeNetwork edge services for multi-environment deployments.
• To learn more about Cloud Service Mesh, see theCloud Service Mesh overview.
• To learn about internet NEGs, seeCloud Service Mesh with internet network endpoint groups.