Service networking for distributed applications in Cross-Cloud Network

Last reviewed 2025-01-30 UTC

This document is part of a design guide series for Cross-Cloud Network.

The series consists of the following parts:

This document describes how to assemble an application from a set of chosen orcreated component services. We recommend you read through the entire documentonce before following the steps.

This document guides you through the following decisions:

  • Whether you create the individual service yourself or consume athird-party service
  • Whether the service is available globally or regionally
  • Whether the service is consumed from on-premises, from other clouds, orfrom neither
  • Whether you access the service endpoint through a shared services VPC ordistribute the endpoints through all the relevant application VPCs

This document guides you through the following steps:

  1. Deciding if your application is global or regional
  2. Choosing third-party managed services or creating and publishing your ownservices
  3. Setting up private access to the service endpoints using either a shared ordedicated mode
  4. Assembling the services into applications to match either a global orregional archetype

Developers define the service networking layer for Cross-Cloud Network. By this stage, administratorshave designed aconnectivitylayer for Cross-Cloud Network that allowsflexibility in the service networking options described in this document. Insome cases, constraints from limited cross-VPC transitivityexist. We describe these constraints when they can affect design decisions.

Decide whether your application is regional or global

Determine if the customers of the application you are creating need a regionalor global deployment archetype. You can achieve regional resiliency by spreadingloads across the zones of a region. You can achieve global resiliency byspreading loads across regions.

Consider the following factors when choosing an archetype:

  • The availability requirements of the application
  • The location where the application is consumed
  • Cost

For details, seeGoogle Cloud deploymentarchetypes.

This design guide discusses how to support the following deployment archetypes:

In a cross-cloud distributed application, different services of that applicationcan be delivered from different cloud service providers (CSPs) or private data centers.To help ensure a consistent resiliency structure, put services hosted in different CSPs into CSP data centers that are geographically near each other.

The following diagram shows a global application stack that's distributedacross clouds and different application services are deployed in different CSPs. Eachglobal application service has workload instances in different regions of the same CSP.

Global application stack distributed acrossclouds.

Define and access application services

To assemble your application, you can use existing third-party managed services,create and host your own application services, or use a combination of both.

Use existing third-party managed services

Decide which third-party managed services you can use for your application. Determine whichones are constructed as regional services or global services. Also,determine which private access options each service supports.

When you know which managed services you can use, you can determine whichservices you need to create.

Create and access application services

Each service is hosted by one or more workload instances that can be accessed asa single endpoint or as a group of endpoints.

The general pattern for an application service is shown in the followingdiagram. The application service is deployed across a collection of workloadinstances. (In this case, a workload instance could be a Compute Engine VM,a Google Kubernetes Engine (GKE) cluster, or some other backend that runs code.)The workload instancesare organized as a set of backends that are associated with a load balancer.

The following diagram shows a generic load balancer with a set of backends.

Load balancer with backends.

To achieve the chosen load distribution and to automate failovers, these groups of endpoints use a frontend load balancer. By using managed instancegroups (MIGs), you can elastically increase or decrease the capacity of theservice by autoscaling the endpoints that form the backend of the loadbalancer. Furthermore, depending on the requirements of the application service,the load balancer can also provide authentication, TLS termination, and otherconnection specific services.

Determine the scope of the service - regional or global

Decide if your service needs and can support regional or global resiliency. A regional software service can be designed for synchronization within a regional latency budget. A global application service can support synchronousfailovers across nodes that are distributed across regions. If your applicationis global, you might want the services supporting it to be global as well. But, ifyour service requires synchronization among its instances to support failover,you must consider the latency between regions. For your situation, you mighthave to rely on regional services with redundancy in the same or nearby regions,thus supporting low-latency synchronization for failover.

Cloud Load Balancing supports endpoints that are hosted eitherwithin a single region or distributed across regions. Thus, you can create aglobal customer-facing layer that speaks to global, regional, or hybrid servicelayers. Choose your service deployments to ensure that dynamic network failover(or load balancing across regions) aligns with the resiliency state andcapabilities of your application logic.

The following diagram shows how a global service that's built from regional loadbalancers can reach backends in other regions, thus providingautomatic failover across regions. In this example, the application logic isglobal and the chosen backend supports synchronization across regions. Each loadbalancer primarily sends requests to the local region, but can failover to remoteregions.

Load balancers with backends in different regions.

  • A global backend is a collection of regional backends that are accessed by oneor more load balancers.
  • Although the backends are global, the frontend of each load balancer is regional.
  • In this architecture pattern, load balancers primarily distribute traffic only within theirregion, but can also balance traffic to other regions when the resources in the local region are unavailable.
  • A set of regional load balancer frontends, each accessible from other regionsand each able to reach backends in other regions, form an aggregate global service.
  • To assemble a global application stack, as discussed inDesign global application stacks,you can use DNS routingand health checks to achieve cross-regional failover of the frontends.
  • The load balancer frontends are themselves accessible from other regions usingglobal access (not shown in diagram).

This same pattern can be used to include published services with globalfailover. The following diagram depicts a published service that usesglobal backends.

Load balancers accessible from different regions.

In the diagram, note that the published service has global failover implementedin the producer environment. The addition of global failover in the consumer environment enables resilience to regional failures in the consumer load balancing infrastructure. Cross-regional failover of the service must be implemented both in the service application logic and in the load balancing design of the service producer. Other mechanisms can be implemented by the serviceproducers.

To determine which Cloud Load Balancing product to use, you must firstdetermine what traffic type your load balancers must handle. Consider thefollowing general rules:

  • Use an Application Load Balancer for HTTP(S) traffic.
  • Use a proxy Network Load Balancer for non-HTTP(S) TCP traffic. This proxy load balanceralso supports TLS offload.
  • Use a passthrough Network Load Balancer to preserve the client source IP address in the header,or to support additional protocols like UDP, ESP, and ICMP.

For detailed guidance on selecting the best load balancer for your use case, seeChoose a loadbalancer.

Services with serverless backends

A service can be defined using serverless backends. The backend in the producerenvironment can be organized in aServerlessNEGas a backend to a load balancer. This service can be published usingPrivate Service Connect by creating a service attachment that'sassociated with the frontend of the producer load balancer. The publishedservice can be consumed through Private Service Connect endpoints orPrivate Service Connect backends.If the service requires producer-initiated connections, you can use aServerless VPC Accessconnector to let Cloud Run, App Engine standard, andCloud Run functions environments send packets to the internal IPv4 addresses ofresources in a VPC network. Serverless VPC Access also supports sendingpackets to other networks connected to the selected VPC network.

Methods for accessing services privately

Your application can consist of managed services provided by Google, third-partyservices provided by outside vendors or peer groups in your organization, andservices that your team develops. Some of those services might be accessible overthe internet using public IP addresses. This section describes the methods youcan use to access those services using the private network. The following servicetypes exist:

  • Google public APIs
  • Google serverless APIs
  • Published managed services from Google
  • Published managed services from vendors and peers
  • Your published services

Keep these options in mind when reading subsequent sections. Depending on howyou allocate your services, you can use one or more of the private accessoptions described.

Note: This guide assumes that all interaction between services happens privately. However, if you are using two external services and those services interact directly, how they interact is outside the scope of this guide.

The organization (or group within an organization) that assembles, publishes,and manages a service is referred to as the service producer. You and yourapplication are referred to as the service consumer.

Some managed services are published exclusively usingprivate servicesaccess.The network designs recommended inInternal connectivity and VPCnetworkingaccommodate services published with private service access andPrivate Service Connect.

For an overview of the options for accessing services privately, seePrivateaccess options for services.

We recommend usingPrivate Service Connectto connect to managed services whenever possible. For more information ondeployment patterns for Private Service Connect, seePrivate Service Connect deploymentpatterns.

There are two types of Private Service Connect, and the differentservices can be published as either type:

Services published as Private Service Connect endpoints can beconsumed directly by other workloads. The services rely on the authentication andresiliency provisioned by the producer of the service. If you want additionalcontrol over the service authentication and resiliency, you can usePrivate Service Connect backends to add a layer of load balancingfor authentication and resiliency in the consumer network.

The following diagram shows services being accessed throughPrivate Service Connect endpoints:

Accessing services through Private Service Connect endpoints.

The diagram shows the following pattern:

  • A Private Service Connect endpoint is deployed in the consumerVPC, which makes the producer service available to VMs andGKE nodes.
  • Both the consumer and producer networks must be deployed in the same region.

The preceding diagram shows endpoints as regional resources. Endpoints arereachable from other regions because ofglobalaccess.

For more information on deployment patterns, seePrivate Service Connect deploymentpatterns.

Private Service Connect backends use a load balancer configuredwith Private Service Connect network endpoint group (NEG)backends. For a list of supported load balancers, seeAboutPrivate Service Connectbackends.

Private Service Connect backends let you create the followingbackend configurations:

  • Customer-owned domains and certificates in front of managed services
  • Consumer-controlled failover between managed services in different regions
  • Centralized security configuration and access control for managed services

In the following diagram, the global load balancer uses aPrivate Service Connect NEG as a backend that establishescommunication to the service provider. No further networking configuration isrequired and the data is carried over Google's SDN fabric.

Global load balancer using network endpoint group.

Most services are designed for connections that the consumer initiates.When services need to initiate connections from the producer, usePrivate Service Connectinterfaces.

A key consideration when deploying private service access orPrivate Service Connect is transitivity.Private Service Connect consumer access points are reachable overNetwork Connectivity Center. Published services are not reachable across aVPC Network Peering connection for eitherPrivate Service Connect or private services access consumeraccess points. In the absence of inter-VPC transitivity for allservice consumer access points, the location of the service access subnet orendpoints in the VPC topology dictates whether you design thenetwork for shared or dedicated service deployment.

Options such as HA VPN and customer managed proxies provide methods to allowinter-VPC transitive communication.

Private Service Connect Endpoints aren't reachable overVPC Network Peering. If you require this type of connectivity, deploy an internal load balancerand Private Service Connect NEG as a backend, as shown in the following diagram:

Using NEGs to provide reachability.

Google APIs can be accessed privately by usingPrivate Service Connect endpoints and backends.The use of endpoints is generally recommended as the GoogleAPI producer provides resiliency and certificate based authentication.

Create a Private Service Connect endpoint in every VPCin which the service needs to be accessed. Because the consumer IP address is a private globalIP address, a single DNS mapping for each service is required, even if there areendpoint instances in multiple VPCs, as shown in the followingdiagram:

Private Service Connect with Google APIs.

Define consumption patterns for published services

Published services can run in a variety of locations: your VPC network, another VPCnetwork, in an on-premises data center, or in the cloud. Regardless of where theservice workload runs, your application consumes those services by using anaccess point, such as one of the following:

  • An IP address in a private services access subnet
  • A Private Service Connect endpoint
  • A VIP for a load balancer using Private Service Connect NEGs

The consumer access points can be shared across networks or dedicated to asingle network. Decide whether to create shared or dedicated consumer accesspoints based on whether your organization delegates the task of creating consumer serviceaccess points to each application group or manages the access to services ina consolidated manner.

The management of service access involves the following activities:

  • Creating the access points
  • Deploying the access points in a services access VPC, which isa VPC that has the appropriate type of reachability
  • Registering the consumer access points' IP addresses and URLs in DNS
  • Managing security certificates and resiliency for the service in theconsumer space, when adding load balancing in front of the consumer accesspoints

For some organizations, it might be viable to assign service access management to a centralteam, while others might be structured to give more independence to each consumeror application team. A byproduct of operating in the dedicated mode is that someof the elements are replicated. For example, a service is registeredwith multiple DNS names by each application group and manages multiple setsof TLS certificates.

The VPC design described inNetwork segmentation andconnectivity for distributed applications inCross-Cloud Networkenables reachability for deploying service access points in either a shared ordedicated mode. Shared consumer access points are deployed in service accessVPCs, which can be accessed from any other VPC orexternal network. Dedicated consumer access points are deployed in applicationVPCs, which can be accessed only from resources within thatapplication VPC.

The main difference between a service accessVPC and an application VPC is the service accesspoint transitive connectivity that a service access VPC enables. Serviceaccess VPCs aren't limited to hosting consumer access points. AVPC can host application resources, as well as shared consumeraccess points. In such a case, the VPC should be configured andhandled as a serviceVPC.

Shared managed services access

For all service consumption methods, includingPrivate Service Connect, ensure that you do the following tasks:

  • Deploy the services consumer access points in a services VPC. Service VPCshave transitive reachability to other VPCs.
  • If the service access VPC is connected with HA VPN,advertise the subnet for the consumer access point as acustom routeadvertisementfrom the Cloud Router that peers to other networks overHA VPN. For Google APIs, advertise the host IP address of the API.
  • Update multicloud firewall rules to allow the private services access subnet.

For private service access specifically, ensure that you can fulfill thefollowing additional requirements:

For serverless service access, ensure that you can fulfill the followingrequirements:

  • Access connector requires a dedicated /28 regular subnet
  • Cloud Router advertises regular subnets by default
  • Create ingress firewall rules to all allow VPC access subnet within the VPC(s)
  • Update multicloud firewall rules to allow the VPC access connector subnet
  • Create ingress firewall rule(s) to allow the private services access subnetinto the application VPC(s)

Dedicated managed services access

Ensure that you do the following tasks:

  • In each application VPC whereaccess is needed, deploy a forwarding rule for the service to create an accesspoint.
  • For private service access, create ingress firewall rule(s) to allow theprivate services access subnet into the application VPC(s).

For serverless service access, ensure that you can fulfill the followingrequirements:

  • Access connector requires a dedicated /28 regular subnet
  • Create ingress firewall rules to allow the VPC Access connector subnetwithin the application VPC(s)

Assemble the application stack

This section describes assembling a regional or global application stack.

Design regional application stacks

When an application follows the regional or multi-regional deploymentarchetypes, use regional application stacks. Regional application stacks can bethought of as a concatenation of regional application service layers. Forexample, a regional web service layer that talks with an application layer inthe same region, that in turn talks to a database layer in the same region, is aregional application stack.

Each regional application service layer uses load balancers to distributetraffic across the endpoints of the layer in that region. Reliability isachieved by distributing the backend resources across three or more zones in theregion.

Application service layers in other CSPs or on-premises datacenters should be deployed in equivalent regions in the external networks. Also,make published services available in the region of the stack. To align theapplication stack within a region, the application service layer URLs have toresolve to the specific load balancer frontend regional IP address. These DNSmappings are registered in the relevant DNS zone for each application service.

The following diagram shows a regional stack with active-standby resiliency:

Regional stack with active-standby resiliency.

A complete instance of the application stack is deployed in each region acrossthe different cloud data centers. When a regional failure occurs on any of theapplication service layers, the stack in the other region takes over delivery ofthe entire application. This failover is done in response to out-of-bandmonitoring of the different application service layers.

When a failure isreported for one of the service layers, the frontend of the application isre-anchored to the backup stack. The application is written to reference a setof regional name records that reflect the regional IP address stack in DNS so that eachlayer of the application maintains its connections within the same region.

Design global application stacks

When an application follows the global application deployment archetype, eachapplication service layer includes backends in multiple regions. Including backends in multiple regions expandsthe resiliency pool for the application service layer beyond a single region andenables automated failover detection and reconvergence.

The following diagram shows a global application stack:

Global stack using a hub project and application project.

The preceding diagram shows a global application assembled from the following components:

  • Services running in on-premises data centers with load balancer frontends. Theload balancer access points are reachable over Cloud Interconnect from thetransit VPC.
  • A transit VPC hosts hybrid connections between theexternal data center and the application VPC.
  • An application VPC that hosts the core application running on workloadinstances. These workload instances are behind load balancers. The loadbalancers are reachable from any region in the network and they can reachbackends in any region in the network.
  • A services VPC that hosts access points for services running in otherlocations, such as in third party VPCs. These service access points arereachable over the HA VPN connection between the services VPC and thetransit VPC.
  • Service producer VPCs that are hosted by other organizations or theprimary organization and applications that run in other locations. The relevantservices are made reachable by Private Service Connect backendsdeployed as global backends to regional load balancers hosted in theservices VPC. The regional load balancers are reachable from any other region.

If you want the created application to be reachable from the internet, you canadd aglobal external Application Load Balancerthat points to the application workloads in the application VPC(not shown in the diagram).

To support a global application stack, we used global backends for eachapplication layer. This allows recovery from a failure of all backend endpointsin one region. Each region has a regional load balancer frontend for eachapplication service layer. When a regional failover occurs, the internal regionalload balancer frontends can be reached across regions, because they use globalaccess. Because the application stack is global,DNS geolocation routingpoliciesare used to select the most appropriate regional frontend for a particularrequest or flow. In case of a frontend failure,DNS healthcheckscan be used to automate the failover from one frontend to another.

Services published using Private Service Connect backendsbenefit fromPrivate Service Connect globalaccess.This feature allows aPrivate Service Connect backend to be reachable from any regionand lessens disruptions from application service layer failures. Thismeans the Private Service Connect backends can be leveraged asglobal backends as described inDetermine the scope of the Service - regionalorglobal.

Provide private access to services hosted in external networks

You might want to publish a local access point for a service hosted in anothernetwork. In these cases, you can use aninternal regional TCP proxy loadbalancerusing hybrid NEGs. You can create a service producer that's hosted on-premises or inother cloud environments that are available to service consumers (clients) in yourVPC network, as shown in the following diagram:

Local access point using hybrid NEGs.

If you want to make the hybrid service available in a VPC network other than theone hosting the load balancer, you can usePrivate Service Connect topublish theservice.By placing a service attachment in front of your internal regional TCP proxyload balancer, you can let clients in other VPC networks reach the hybridservices running on-premises or in other cloud environments.

In a cross-cloud environment, the use of a hybrid NEG allows for secureapplication-to-application communication.

When a different organization publishes an application service, use a hybrid NEGto extend private access abstractions for that service. Thefollowing diagram illustrates this scenario:

Hybrid NEGs in front of services in other networks.

In the preceding diagram, the application service layer is fully composed in theneighboring CSP, which is highlighted in the parts of the diagram that aren'tgrayed out. The hybrid load balancers are used in conjunction withPrivate Service Connect service attachments as a mechanism topublish the external application service for private consumption withinGoogle Cloud. The hybrid load balancers with hybrid NEGs andPrivate Service Connect service attachments are in a VPC that'spart of a service producer project. This service producer project might usually be a different VPC thanthe transit VPC, because it is within the administrative scope of the producerorganization or project, and therefore separate from the common transitservices. The producer VPC doesn't need to be connected over VPCpeering or HA VPN with the consumer VPC (which is the ServicesShared VPC in the diagram).

Centralize service access

Service access can be centralized into a VPC network and accessedfrom other application networks. The following diagram shows the common patternthat enables the centralization of the access points:

Dedicated services VPC.

The following diagram shows all servicesbeing accessed from a dedicated services VPC:

Dedicated services VPC with centralized load balancers.

When services are frontended with application load balancers, you can consolidateonto fewer load balancers by using URL mapsto steer traffic for different service backends instead of using different load balancers for each service.In principle, an application stack could be fully composedusing a single application load balancer plus service backends and appropriate URL mappings.

In this implementation, you must use hybrid NEGs across VPCs for most backendtypes. The exception is a Private Service Connect NEG orbackend, as described inExplicit Chaining of Google Cloud L7 Load Balancers with Private Service Connect.

The use of hybrid NEGs across VPCs comes at the expense of foregoingautohealing and autoscaling for the backends. Published services already havea load balancer in the producer tenant that provides autoscaling. Therefore, youonly run into the limitations of the hybrid NEGs if youcentralize the load balancing function for service layers being composednatively rather than consumed from publication.

When using this service networking pattern, remember that the services areconsumed through an additional layer of load balancing.

Private Service Connect service endpoints are reachable across NCC spoke VPCs.

The centralized mode adds a layer of load balancing on the consumer side of theservice. When you use this mode, you also need to manage certificates,resiliency, and additional DNS mappings in the consumer project.

Other considerations

This section contains considerations for common products and services notexplicitly covered in this document.

GKE control plane considerations

The GKE control plane is deployed in a Google managedtenant project that's connected to the customer's VPC using VPC Network Peering. BecauseVPC Network Peering isn't transitive, direct communication to the control plane over ahub and spoke VPC peered networking topology isn't possible.

When considering GKE design options, such as centralized ordecentralized, direct access to the control plane from multicloud sources is akey consideration. If GKE is deployed in a centralized VPC,access to the control plane is available across clouds and withinGoogle Cloud. However, if GKE is deployed indecentralized VPCs, direct communication to the control plane isn't possible.If an organization's requirements necessitate access to theGKE control plane in addition to adopting the decentralizeddesign pattern, network administrators can deploy aconnectagentthat acts as a proxy, thus overcoming the non-transitive peering limitation tothe GKE control plane.

Security - VPC Service Controls

For workloads involving sensitive data, useVPC Service Controlsto configure service perimeters around your VPC resources and Google-managedservices, and control the movement of data across the perimeter boundary. UsingVPC Service Controls, you can group projects and your on-premises networkinto a single perimeter that prevents data access through Google-managedservices. VPC Service Controlsingress and egressrulescan be used to allow projects and services in different service perimeters tocommunicate (includingVPC networks that aren't inside theperimeter).

For recommended deployment architectures, a comprehensive onboarding process,and operational best practices, see theBest practices forVPC Service Controls forenterprisesandSecurity FoundationsBlueprint.

DNS for APIs/Services

Service producers canpublish services by usingPrivate Service Connect.The service producer can optionally configure a DNS domain name to associatewith the service. If a domain name is configured, and aservice consumercreates an endpoint that targets thatservice,Private Service Connect and Service Directoryautomatically create DNS entries for the service in a private DNS zone in theservice consumer's VPC network.

What's next

Contributors

Authors:

Other contributors:

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-01-30 UTC.