This document provides a reference architecture that you can use to deploy aCross-Cloud Networkhub-and-spoke network topology in Google Cloud. This network design enables thedeployment of software services across Google Cloud and external networks,like on-premises data centers or other Cloud Service Providers (CSPs).

This design supports multiple external connections, multiple services-accessVirtual Private Cloud (VPC) networks, and multiple workload VPCnetworks.

The intended audience for this document is network administrators who buildnetwork connectivity and cloud architects who plan how workloads are deployed.The document assumes that you have a basic understanding of routing and internetconnectivity.

Architecture

The following diagram shows a high-level view of the architecture of thenetworks and the four packet flows that this architecture supports.

The architecture contains the following high-level elements:

The following diagram shows a detailed view of the architecture that highlightsthe four connections among the networks:

Connections descriptions

This section describes the four connections that are shown in the precedingdiagram.

Connection 1: Between external networks and the transit VPC network

This connection between external networks and transit VPCnetworks happens over Cloud Interconnect orHA VPN. Routes are exchanged by using BGP between theCloud Routers in the transit VPC network and the externalrouters in the external network.

• Routers in external networks announce the routes for external subnets to thetransit VPC Cloud Routers. In general, externalrouters in a given location announce routes from the same external locationas more preferred than routes for other external locations. The preferenceof the routes can be expressed by using BGP metrics and attributes.
• Cloud Routers in the transit VPC network advertiseroutes for prefixes in Google Cloud's VPCs to the externalnetworks. These routes must be announced by using Cloud Router customroute announcements.

Connection 2: Between transit VPC networks and services-access VPC networks

This connection between transit VPC networks and services-accessVPC networks happens over HA VPN withseparate tunnels for each region. Routes are exchanged by using BGP between theregional Cloud Routers in the transit VPC networks andthe services-access VPC networks.

• Transit VPC HA VPNCloud Routers announce routes for external network prefixes,workload VPCs, and other services-access VPCsto the services-access VPC Cloud Router. Theseroutes must be announced by using Cloud Router custom routeannouncements.
• The services-access VPC network announces its subnets and thesubnets of any attached managed services VPC networks to thetransit VPC network. Managed services VPCroutes and the services-access VPC subnet routes must beannounced by using Cloud Router custom route announcements.

Connection 3: Between transit VPC networks and workload VPC networks

This connection between transit VPC networks and workloadVPC networks is implemented over VPC peering.Subnets and prefix routes are exchanged by using VPC peeringmechanisms. This connection allows communication between the workloadVPC networks and the other networks that are connected to the transitVPC network, including the external networks and theservices-access VPC networks.

• The transit VPC network uses VPC Network Peering toexport custom routes. These custom routes include all of the dynamic routesthat have been learned by the transit VPC network.The workload VPC networks import those custom routes.
• The workload VPC network automatically exports subnets to thetransit VPC network. No custom routes are exported from theworkload VPCs to the transit VPC.

Connection 4: Between workload VPC networks

• Workload VPC networks can be connected together by usingNCC VPC spokes. This is an optionalconfiguration. You can omit it if you don't want workload VPCnetworks to communicate with each other.

Traffic flows

The following diagram shows the four flows that are enabled by this referencearchitecture.

The following table describes the flows in the diagram:

Products used

This reference architecture uses the following Google Cloud products:

• Virtual Private Cloud (VPC): A virtual system that provides global, scalablenetworking functionality for your Google Cloud workloads. VPC includesVPC Network Peering, Private Service Connect, private services access, andShared VPC.
• Network Connectivity Center: An orchestration framework that simplifiesnetwork connectivity among spoke resources that are connected to a central management resourcecalled ahub.
• Cloud Interconnect: A service that extends your external network to theGoogle network through a high-availability, low-latency connection.
• Cloud VPN: A service that securely extends your peer network to Google'snetwork through an IPsec VPN tunnel.
• Cloud Router: A distributed and fully managed offering that provides Border Gateway Protocol(BGP) speaker and responder capabilities. Cloud Router works with Cloud Interconnect,Cloud VPN, and Router appliances to create dynamic routes in VPCnetworks based on BGP-received and custom learned routes.

Design Considerations

This section describes design factors, best practices, and design recommendations that you should consider when you use this reference architecture to develop a topology that meets your specific requirements for security, reliability, and performance.

Security and compliance

The following list describes the security and compliance considerations for thisreference architecture:

• For compliance reasons, you might want to deploy workloads in a singleregion only. If you want to keep all traffic in a single region, you can usea 99.9% topology. For more information, seeEstablish 99.9% availability forDedicated InterconnectandEstablish 99.9% availability forPartner Interconnect.
• Use Cloud Next Generation Firewall to secure traffic that enters and leaves theservices-access and workload VPC networks. To secure trafficthat passes between external networks and the transit network, you need touse external firewalls or NVA firewalls.
• Enable logging and monitoring as appropriate for your traffic and complianceneeds. You can useVPC Flow Logs to gaininsights into your traffic patterns.
• UseCloud IDS togather additional insight into your traffic.

Reliability

The following list describes the reliability considerations for this referencearchitecture:

• To get 99.99% availability for Cloud Interconnect, you must connectto two different Google Cloud regions.
• To improve reliability and minimize exposure to regional failures, you candistribute workloads and other cloud resources across regions.
• To handle your expected traffic, create a sufficient number of VPN tunnels.Individual VPN tunnels havebandwidthlimits.

Performance optimization

The following list describes the performance considerations for this referencearchitecture:

• You might be able to improve network performance by increasing the maximumtransmission unit (MTU) of your networks and connections. For moreinformation, seeMaximum transmission unit.
• Communication between the transit VPC and workload resourcesis over VPC Network Peering, which provides full-line-rate throughputfor all VMs in the network at no additional cost. ConsiderVPC Network Peering quotas and limitswhen you plan your deployment. You have several choices in connecting yourexternal network to the transit network. For more information aboutbalancing cost and performance considerations, seeChoosing aNetwork Connectivityproduct.

Deployment

The architecture in this document creates three sets of connections to a central transitVPC network plus a different connection among workloadVPC networks. After all of the connections are fully configured,all of the networks in the deployment can communicate with all other networks.

This deployment assumes that you are creating connections between the external andtransit networks in two regions. Workload subnets can be in any region, however.If you are placing workloads in one region only, you only need to create subnetsin that region.

To deploy this reference architecture, complete the following tasks:

1. Identify regions to place connectivity and workloads
2. Create your VPC networks and subnets
3. Create connections between external networks and your transit VPC network
4. Create connections between your transit VPC network and services-access VPC networks
5. Create connections between your transit VPC network and workload VPC networks
6. Connect your workload VPC networks
7. Test connectivity to workloads

Identify regions to place connectivity and workloads

In general, you want to place connectivity and Google Cloud workloads inclose proximity to your on-premises networks or other cloud clients. For moreinformation about placing workloads, seeGoogle Cloud RegionPicker andBest practices forCompute Engine regionsselection.

Create your VPC networks and subnets

To create your VPC networks and subnets, complete the followingtasks:

1. Create or identify the projects where you will create your VPCnetworks. For guidance, seeNetwork segmentation and projectstructure.If you intend to useShared VPC networks,provision your projects as Shared VPC hostprojects.
2. Plan your IP address allocations for your networks. You can preallocate andreserve your ranges bycreating internalranges. Allocating address blocks thatcan be aggregated makes later configuration and operations morestraightforward.
3. Create a transit network VPC with global routing enabled.
4. Create service VPC networks. If you will have workloads inmultiple regions, enable global routing.
5. Create workload VPC networks. If you will have workloads inmultiple regions, enable global routing.

Create connections between external networks and your transit VPC network

This section assumes connectivity in two regions and assumes that the externallocations are connected and can fail over to each other. It also assumes thatthere is a preference for clients in external location A to reach services inregion A, and so on.

1. Set up the connectivity between the external networks and your transitnetwork. For an understanding of how to think about this, seeExternal andhybridconnectivity.For guidance on choosing a connectivity product, seeChoosing aNetwork Connectivityproduct.
2. Configure BGP ineach connected region as follows:
   • Configure the router in the given external location as follows:
     • Announce all subnets for that external location by using the same BGPMED on both interfaces, such as 100. If both interfaces announce thesame MED, then Google Cloud can use ECMP to load balancetraffic across both connections.
     • Announce all subnets from the other external location by using alower-priority MED than that of the first region, such as 200.Announce the same MED from both interfaces.
   • Configure the external-facing Cloud Router in the transitVPC of the connected region as follows:
     • Set your Cloud Router ASN to be 16550.
     • Usingcustom routeadvertisements,announce all subnet ranges from all regions over bothexternal-facing Cloud Router interfaces. Aggregate them ifpossible. Use the same MED on both interfaces, such as 100.

Create connections between your transit VPC network and services-access VPC networks

To provide transitive routing between external networks and the services-accessVPC and between workload VPCs and theservices-access VPC, the services-access VPC usesHA VPN for connectivity.

1. Estimate how much traffic needs to travel between the transit andservices-access VPCs in each region. Scale your expectednumber of tunnels accordingly.
2. Configure HA VPN between the transit VPCand the services-access VPC in region A by using the instructionsinCreate HA VPN gateways to connectVPCnetworks. Create adedicated HA VPN Cloud Router in the transitnetwork. Leave the external-network-facing router for external networkconnections.
   • Transit VPC Cloud Router configuration:
     • To announce external-network and workload VPC subnets tothe services-access VPC, usecustom routeadvertisementson the Cloud Router in the transit VPC.
   • Services-access VPC Cloud Router configuration:
     • To announce services-access VPC subnets to the transitVPC, use custom route advertisements on the services-accessVPC Cloud Router.
     • If you useprivate servicesaccess to connect a managedservices VPC to the services-accessVPC, use custom routes to announce those subnets aswell.
3. If you connect a managed services VPC to the services-accessVPC by usingprivate servicesaccess, after the VPC Network Peeringconnection is established, update the services-access VPCside of the VPC Network Peering connection to export custom routes.

Create connections between your transit VPC network and workload VPC networks

Create VPC Network Peering connectionsbetween your transit VPC and each of your workloadVPCs:

• EnableExport custom routes for the transit VPC side ofeach connection.
• EnableImport custom routes for the workload VPC sideof each connection.
• In the default scenario, only the workload VPC subnet routesare exported to the Transit VPC. You don't need to exportcustom routes from the workload VPCs.

Connect your workload VPC networks

Connect the workload VPC networks together by usingNCC VPCspokes.Make all spokes part of the same NCC spoke peer group. Use a corepeer group to allow full mesh communication between the VPCs.

The NCC connection announces specific routes among the workloadVPC networks. Traffic between these networks follows thoseroutes.

Test connectivity to workloads

If you have workloads already deployed in your VPC networks, testaccess to them now. If you connected the networks before deploying workloads,you can deploy them now and test.

What's next

• Learn more about the Google Cloud products used in this design guide:
  • VPC networks
  • VPC Network Peering
  • Cloud Interconnect
  • HA VPN
• For more reference architectures, diagrams, and best practices, explore theCloud Architecture Center.

Contributors

Authors:

• Deepak Michael | Networking Specialist Customer Engineer
• Victor Moreno | Product Manager, Cloud Networking
• Osvaldo Costa | Networking Specialist Customer Engineer

Other contributors:

• Mark Schlagenhauf | Technical Writer, Networking
• Ammett Williams | Developer Relations Engineer
• Ghaleb Al-habian | Network Specialist