This guide introduces best practices and typical enterprise architectures forthe design ofVirtual Private Cloud (VPC) with Google Cloud. This guide is for cloud network architects andsystem architects who are already familiar with Google Cloud networkingconcepts.

General principles and first steps

Identify decision makers, timelines, and pre-work

As a first step in your VPC network design, identify the decision makers, timelines,and pre-work necessary to ensure that you can address stakeholderrequirements.

Stakeholders might include application owners, security architects, solutionarchitects, and operations managers. The stakeholders themselves might changedepending on whether you are planning your VPC network for aproject, a line of business, or the entire organization.

Part of the pre-work is to get the team acquainted with concepts andterminology around VPC network design. Useful documents include thefollowing:

• Resource Manager documentation
• Identity and Access Management (IAM) documentation
• Virtual Private Cloud documentation

Consider VPC network design early

Make VPC network design an early part of designing yourorganizational setup in Google Cloud. It can be disruptive to yourorganization if you later need to change fundamental things such as how yournetwork is segmented or where your workloads are located.

Different VPC network configurations can have significantimplications for routing, scale, and security. Careful planning and deepunderstanding of your specific considerations helps you to create a solidarchitectural foundation for incremental workloads.

Keep it simple

Keeping the design of your VPC network topology simple is the best way to ensure amanageable, reliable, and well-understood architecture.

Use clear naming conventions

Make your naming conventions simple, intuitive, and consistent. This ensuresthat administrators and end users understand the purpose of each resource, whereit is located, and how it is differentiated from other resources.

Commonly accepted abbreviations of long words help with brevity. Using familiarterminology where possible helps with readability.

Consider the components illustrated in the following example when establishingyour naming conventions:

• Company name: Acme Company:acmeco
• Business unit: Human Resources:hr
• Application code: Compensation system:comp
• Region code: northamerica-northeast1:na-ne1, europe-west1:eu-we1
• Environment codes:dev,test,uat,stage,prod

In this example, the development environment for the human resourcesdepartment's compensation system is namedacmeco-hr-comp-eu-we1-dev.

For other common networking resources, consider patterns like these:

• VPC network
  syntax:{company name}-{description(App or BU)-label}-{environment-label}-{seq#}
  example:acmeco-hr-dev-vpc-1

• Subnet
  syntax:{company-name}-{description(App or BU)-label}-{region-label}-{environment-label}-{subnet-label}
  example:acmeco-hr-na-ne1-dev-vpc-1-subnet-1
  note: If a subnet contains resources in one zone only, you can use "zone-label" instead of "region-label."

• Firewall rule
  syntax:{company-name}-{description(App or BU)-label}{source-label}-{dest-label}-{protocol}-{port}-{action}
  example:acmeco-hr-internet-internal-tcp-80-allow-rule

• IP route
  syntax:{priority}-{VPC-label}-{tag}-{next hop}
  example:1000-acmeco-hr-dev-vpc-1-int-gw

Addresses and subnets

Use custom mode VPC networks

Though auto mode networks can be useful for early exploration, custom modeVPC networks are better suited for most production environments.

We recommend that enterprises use VPC networks in custom modefrom the beginning for the following reasons:

• Custom mode VPC networks better integrate into existing IPaddress management schemes. Because all auto mode networks use the sameset of internal IP ranges, auto mode IP ranges might overlap when connectedwith your on-premises corporate networks.
• You can't connect two auto mode VPC networks together usingVPC Network Peering because their subnets use identical primary IP ranges.
• Auto mode subnets all have the same name as the network. You can chooseunique, descriptive names for custom mode subnets, makingyour VPC networks more understandable and maintainable.
• When a new Google Cloud region is introduced, auto modeVPC networks automatically get a new subnet in that region.Custom mode VPC networks only get new subnets if you specifythem. This can be important for both sovereignty and IP address managementreasons.

If it has no resources, you candelete thedefaultnetwork. You cannot delete a VPC network until you haveremoved all resources, including Virtual Machine (VM) instances, that depend onit.

For more details of the differences between auto mode and custom modeVPC networks, see theVPC network overview.

Group applications into fewer subnets with larger address ranges

Conventionally, some enterprise networks are separated into many small addressranges for a variety of reasons. For example, this might have been done toidentify or isolate an application or keep a small broadcast domain.

However, we recommend that you group applications of the same type into fewer,more manageable subnets with larger address ranges in the regions you want tooperate.

Unlike other networking environments in which a subnet mask is used,Google Cloud uses a software-defined networking (SDN) approach to providea full mesh of reachability between all VMs in the global VPC network.The number of subnets does not affect routing behaviour.

You can use service accounts or network tags to apply specific routing policiesor firewall rules. Identity in Google Cloud is not based solely on thesubnet IP address.

Some VPC features—includingCloud NAT,Private Google Access,VPC Flow Logs,andalias IP ranges—areconfigured per subnet. If you need more fine-grained control of these features,use additional subnets.

Single VPC network and Shared VPC

Start with a single VPC network for resources that have common requirements

For many simple use cases, a single VPC network provides the features that you need,while being easier to create, maintain, and understand than the more complexalternatives. By grouping resources with common requirements and characteristicsinto a single VPC network, you begin to establish theVPC network border as the perimeter for potential issues.

For an example of this configuration, see thesingle project, single VPC network reference architecture.

Factors that might lead you to create additional VPC networksinclude scale, network security, financial considerations, operationalrequirements, and identity and access management (IAM).

Use Shared VPC for administration of multiple working groups

For organizations with multiple teams,Shared VPC provides an effective tool to extend the architectural simplicity of a singleVPC network across multiple working groups.The simplest approach is to deploy a single Shared VPC host project with asingle Shared VPC network and then attach team service projects to the host project network.

In this configuration, network policy and control for all networking resourcesare centralized and easier to manage. Service project departments can configureand manage non-network resources, enabling a clear separation ofresponsibilities for different teams in the organization.

Resources in those projects can communicate with each other more securely andefficiently across project boundaries using internal IP addresses. You canmanage shared network resources—such as subnets, routes, and firewalls—from acentral host project, so you can enforce consistent network policies across theprojects.

For an example of this configuration, see theSingle host project, multiple service projects, single Shared VPC reference architecture.

Grant the network user role at the subnet level

The centralized Shared VPC administrator can grant members the network user(networkUser)role either at a subnet level, for fine-grained service-project authorization,or for all subnets at the host project level.

Following the principle of least privilege, we recommend granting the networkuser role at the subnet level to the associated user, service account, orgroup.

Because subnets are regional, this granular control lets you specify whichregions each service project can use to deploy resources.

With Shared VPC architectures, you also have the flexibility to deploy multipleShared VPC host projects within your organization. Each Shared VPC host projectcan then accommodate a single or multiple Shared VPC networks. In thisconfiguration, different environments can enforce different policyconcerns.

For more information, seeIAM roles for networking.

Use a single host project if resources require multiple network interfaces

If you have a service project that needs to deploy resources to multipleisolated VPC networks—for example, VM instances with multiplenetwork interfaces—your host project must contain all of the VPCnetworks that provide the services. This is because a service project isallowed to attach to only one host project.

Use multiple host projects if resource requirements exceed the quota of a single project

In cases where the aggregate resource requirements of all VPC networks can't be metwithin a project's quota, use an architecture with multiple host projects with asingle Shared VPC network per host project, rather than a single host project withmultiple Shared VPC networks. It's important to evaluate your scale requirements,because using a single host project requires multiple VPC networks in the host project, andquotas are enforced at the project level.

For an example of this configuration, see theMultiple host projects, multiple service projects, multiple Shared VPC reference architecture.

Use multiple host projects if you need separate administration policies for each VPC network

Because each project has its own quota, use a separate Shared VPC hostproject for every VPC network to scale aggregate resources. This allows eachVPC network to have separate IAM permissions for networking and security management,because IAM permissions are also implemented at the project level. For example,if you deploy two VPC networks (VPC network A and VPC network B) into the same host project, thenetwork administrator(networkAdmin)role applies to both VPC networks.

Deciding whether to create multiple VPC networks

Create a single VPC network per project to map VPC resource quotas to projects

Quotas are constraints applied at the project or network level. All resourceshave an initial default quota meant to protect you from unexpected resource usage.However, many factors might lead you to want more quota. For most resources,you canrequest additional quota.

We recommend creating a single VPC network per project if you expect to grow beyond thedefault VPC resource quotas. This makes it easier to map project-level quotaincreases to each VPC network rather than to a combination of VPC networks in the same project.

Limits are designed to protect systemresources in aggregate. Limits generally can't be raised easily, althoughGoogle Cloud support and sales teams can work with you to increase somelimits.

SeeVPC resource quotas and limits for current values.

Google Support can increase some scaling limits, but there might be times whenyou need to build multiple VPC networks to meet your scaling requirements. If your VPC networkhas a requirement to scale beyond the limits, discuss your case withGoogle Cloud sales and support teams about the best approach for yourrequirements.

Create a VPC network for each autonomous team, with shared services in a common VPC network

Some large enterprise deployments involve autonomous teams that eachrequire full control over their respective VPC networks. You can meet this requirementby creating a VPC network for each business unit, with shared services in a common VPC network(for example, analytic tools, CI/CD pipeline and build machines, DNS/Directoryservices).

Create VPC networks in different projects for independent IAM controls

A VPC network is a project-level resource with fine-grained, project-levelidentity and access management (IAM) controls,including the following roles:

• networkAdmin
• securityAdmin
• networkUser
• networkViewer

By default, IAM controls are deployed at the project level and each IAMrole applies to all VPC networks within the project.

If you require independent IAM controls per VPC network, create your VPC networks in differentprojects.

If you require IAM roles scoped to specific Compute Engine resources such asVM instances, disks, and images, useIAM policies for Compute Engine resources.

Isolate sensitive data in its own VPC network

For companies that deal with compliance initiatives, sensitive data, or highlyregulated data that is bound by compliance standards such as HIPAA or PCI-DSS,further security measures often make sense. One method that can improve securityand make it easier to prove compliance is to isolate each of these environmentsinto its own VPC network.

Connecting multiple VPC networks

Choose the VPC connection method that meets your cost, performance, and security needs

The next step after deciding to implement multiple VPC networks is connecting thoseVPC networks. VPC networks are isolated tenant spaces within Google'sAndromeda SDN, but there are several ways that you can enable communication between them. Thesubsequent sections provide best practices for choosing a VPC connection method.

Theadvantages and disadvantages of each are summarized in the following table, andsubsequent sections provide best practices for choosing a VPC connectionmethod.

Use NCC VPC spokes

We recommend that you useNCC VPC spokes when you need to connect VPC networks together. NCC VPC spokes allow for address re-use across VPCs in the same project and organization or in a different project and organization.

NCC VPC spokes is the preferred method for connecting VPC networks for the following reasons:

• The data plane is distributed, so there is no gateway bottleneck. Traffic travels across VPC networks as if the VMs were in the same VPC network.
• Inter-network connectivity across different organizations. Networks include VPCs as well as external networks.
• Up to 250 VPC networks per hub.
• Transitive reachability across VPCs of service access points.
• Integrated inter-VPC Cloud NAT to enable IP address re-use across VPCs.
• Defined network reachability rules using pre-set topologies and prefix filters.

Use VPC Network Peering if you need to insert NVAs or if your application doesn't support Private Service Connect

We recommend that you use VPC Network Peering if you need to insert network virtual appliances (NVAs), such as firewall VMs. You might need to insert NVAs for traffic that traverses multiple VPC networks or for private connectivity to services that aren't published by using Private Service Connect.

When you use VPC Network Peering, ensure that the totals of the resourcesneeded for all directly connected peers don't exceed thelimits on VM instances, number of peeringconnections, and internal forwarding rules.

VPC Network Peering enables two VPC networks to connect witheach other internally over Google's SDN, whether or not they belong to the sameproject or the same organization. VPC Network Peering merges the controlplane and flow propagation between each peer, allowing the same forwardingcharacteristics as if all the VMs were in the same VPC network.When VPC networks are peered, all subnets, alias IP ranges, andinternal forwarding rules are accessible, and each VPC networkmaintains its own distributed firewall. VPC Network Peering is nottransitive.

Use external routing if you don't need private IP address communication

If you don't need private IP address communication, you can use externalrouting with external IP addresses or a NAT gateway.

When a VPC network is deployed, a route to Google's default internet gateway isprovisioned with a priority of 1000. If this route exists, and a VM is given anexternal IP address, VMs can send outbound (egress) traffic through Google'sinternet gateway. You can also deploy services behind one of Google's manypublic load-balancing offerings, which allows the services to be reachedexternally.

Externally addressed VMs communicate with each other privately over Google'sbackbone, regardless of regionandNetwork Service Tiers.Use Google Cloud firewall rules to control external inbound (ingress) traffic toyour VMs.

External routing is a good option for scaling purposes, but it's important tounderstand how public routing affects costs. For details, see theNetwork pricing documentation.

Use Cloud VPN to connect VPC networks that host service access points that are not transitively reachable over NCC

HA VPN provides a managed service to connect VPC networks by creatingIPsec tunnels between sets of endpoints. If you configure your Cloud Routerswith custom advertisement mode, you can enable transitive routing across VPC networksand hub-and-spoke topologies as described later in this document.Using Network Connectivity Center, you can use HA VPN tunnels as atransit network between on-premises networks, as explained in theCloud VPN documentation.

Cloud VPN does not support large MTU. For details, seeMTU considerations.

Use virtual appliances to control traffic between VPC networks through a cloud device

Multiple network interface VMs are common for VPC networks that requireadditional security or services between them, because multiple network interface VMs enable VMinstances to bridge communication between VPC networks.

To deploy a VM into multiple VPC networks, you must have theappropriate IAM permission for each VPC network to which the VM connects.

When you deploy a multi-NIC VM between VPC networks, remember that thesubnet IP ranges of the interfaces must not overlap.

Service connectivity

Private Service Connectallowsconsumers to accessservices privately from inside theirVPC network without the need for a network oriented deploymentmodel. Similarly, it letsproducers host these services in their ownseparate VPC networks, and they can offer a private connection to their consumers within the same organization or across organizations.Private Service Connect enables connectivity to first-party and third-partymanaged services, which eliminates the needs for subnet allocation for privateservices access and VPC Network Peering.

Private Service Connect offers a service-centric security model with the following benefits:

• No shared dependencies
• Explicit authorization
• Line-rate performance

Private Service Connect is available in different types that provide different capabilities and modes of communication:

• Private Service Connectendpoints: Endpoints aredeployed by using forwarding rules that provide the consumer an IP addressthat is mapped to the Private Service Connect service.
• Private Service Connectbackends: Backends are deployedby using network endpoint groups (NEGs) that let consumers direct traffic totheir load balancer before the traffic reaches a Private Service Connectservice. If the backends are deployed with an HTTPS load balancer, they cansupport certificates.
• Private Service Connectinterfaces: Interfaceslet the consumer and producer originate traffic, which enablesbidirectional communication. Interfaces can be used in the same VPCnetwork as endpoints and backends.

An alternative to Private Service Connect isprivate servicesaccessthat allows consumers to connect the producer services throughVPC Network Peering. When you use private services access, we recommend thatyou consider IP allocation for each producer service, IP overlap, and sharedquota.

Hybrid design: connecting an on-premises environment

After you have identified the need forhybrid connectivity and have chosen a solution that meets your bandwidth, performance, and securityrequirements, consider how to integrate it into your VPC design.

UsingShared VPC alleviates the need for each project to replicate the same solution. Forexample, when you integrate a Cloud Interconnect solution into aShared VPC, all VMs—regardless of region or service project—can access theCloud Interconnect connection.

Use dynamic routing when possible

Dynamic routing is available on all hybrid solutions, includingHA VPN, Classic VPN, Dedicated Interconnect, andPartner Interconnect. Dynamic routing uses Google'sCloud Router as a Border Gateway Protocol (BGP) speaker to provide dynamicExternal BGP (eBGP) routing.

The Cloud Router is not in the data plane; it only creates routes inthe SDN.

Dynamic routing does not use tags, and the Cloud Router neverre-advertises learned prefixes.

You can enable either of the Cloud Router's two modes, regional orglobal, on each VPC network. If you choose regional routing, the Cloud Routeronly advertises subnets that co-reside in the region where theCloud Router is deployed. Global routing, on the other hand, advertisesall subnets of the given VPC network, regardless of region, but does penalize routes that are advertisedand learned outside of the region. This maintains symmetry within the region byalways preferring a local interconnect, and is calculated by adding a penaltymetric (MED) equal to 200 + TTL in milliseconds between regions.

Static routing

Static routing is only available on Classic VPN. Static routing offersthe ability to set a next-hop route pointing at a Cloud VPN tunnel.

By default, a static route applies to all VMs in the network regardless of region. Staticrouting also lets network administrators selectively set which VMs the routeapplies to by using instance tags, which can be specified when you create aroute.

Static routes apply globally within the VPC network, with the same route priority aseach other. Therefore, if you have multiple tunnels in multiple regions to thesame prefix with the same priority, a VM will use 5-tuple hash-based ECMP acrossall tunnels. To optimize this setup, you can create a preferred in-region routeby referencing instance tags for each region and creating preferred routesaccordingly.

If you don't want outbound (egress) traffic to go through Google's defaultinternet gateway, you can set a preferred default static route to send alltraffic back on-premises through a tunnel.

Use a connectivity VPC network to scale a hub-and-spoke architecture with multiple VPC networks

If you need to scale a hub-and-spoke architecture with multiple VPC networks,configure centralized hybrid connectivity in one or more dedicated VPC networks,then add the hybrid connections and all of the VPC spokes to a NCC hub.You'll need to enableroute exchange with VPCspokes.This configuration allows static routes or dynamically learned routes to be exported to VPC spokes inorder to provide centralized configuration and scale to your VPC network design.

The following diagram illustrates a centralized hybrid connectivity designusing NCC:

Alternatively, you can use VPC Network Peering and custom advertised routesto provide access to shared hybrid connections, if you won't exceed resourcelimits and you require the use of NVAs.

The following diagram illustrates centralized hybrid connectivity withVPC Network Peering custom routes:

This centralized design is in contrast to conventional hybrid connectivity deployment, which usesVPN tunnels or VLAN attachments in each individual VPC network.

Network security

Google Cloud provides robust security features across its infrastructureand services, from the physical security of data centers and custom securityhardware to dedicated teams of researchers. However, securing yourGoogle Cloud resources is a shared responsibility. You must takeappropriate measures to help ensure that your apps and data are protected.

Identify clear security objectives

Before evaluating either cloud-native or cloud-capable security controls, startwith a set of clear security objectives that all stakeholdersagree to as a fundamental part of the product. These objectives should emphasizeachievability, documentation, and iteration, so that they can be referenced andimproved throughout development.

Limit external access

When you create a Google Cloud resource that uses a VPC network, youchoose a network and subnet where the resource resides. The resource is assignedan internal IP address from one of the IP ranges associated with the subnet.Resources in a VPC network can communicate among themselves through internal IPaddresses if firewall rules permit.

Limit access to the internet to only those resources that need it. Resourceswith only a private, internal IP address can still access many Google APIs andservices through Private Service Connect orPrivate Google Access. Private access enablesresources to interact with key Google and Google Cloud services whileremaining isolated from the public internet.

Additionally, useorganizational policies to further restrict which resources are allowed to use external IP addresses.

To allow VMs to access the internet, use Secure Web Proxy if the traffic can bedownloaded over HTTP(S) and you want to implement identity controls. Otherwise,use Cloud NAT.

Define service perimeters for sensitive data

For workloads involving sensitive data, useVPC Service Controls to configure service perimeters around your VPC resources and Google-managedservices and control the movement of data across the perimeter boundary. UsingVPC Service Controls, you can group projects and your on-premises network intoa single perimeter that prevents data access through Google-managed services.Service perimeters can't contain projects from different organizations, butyou can use perimeter bridges to allow projects and services in differentservice perimeters to communicate.

Manage traffic with Google Cloud firewall rules when possible

Google Cloud VPC includes a stateful firewall that is horizontallyscalable and applied to each VM in a distributed manner. See theCloud NGFW overview for details.

Google Cloud Marketplace features a large ecosystem of third-party solutions,including VMs that do the following: provide advanced security, such asprotection from information leakage, application exploits, and escalation ofprivileges; detect known and unknown threats; and apply URL filtering. There arealso operational benefits to having a single vendor implement policy acrosscloud service providers and on-premises environments.

Traffic is typically routed to these VMs by specifying routes, either withthe same priority (to distribute the traffic using a 5-tuple hash) or withdifferent priorities (to create a redundant path), as shown in the multiplepaths to the Dev-subnet in the following diagram.

Most solutions require multiple network interface VMs.

Scale is also an important consideration when deploying third-party solutionsinto your VPC network for the following reasons:

• Limits: Most VM-based appliances must be inserted into the datapath. This requires a multiple network interface VM that bridges multiple VPC networks that reside inthe same project. Because VPC resource quotas are set at the project level, theaggregate resource needs across all VPC networks can become limiting.
• Performance: Introducing a single VM-based chokepoint into the fullyhorizontal scalability attributes of a VPC network goes against cloud designmethodologies. To mitigate this, you can place multiple network virtualappliances (NVAs) into a managed instance group behind an internal passthrough Network Load Balancer.

To account for these factors in high-scale requirement architectures, pushsecurity controls to your endpoints. Start by hardening your VMs and using Google Cloudfirewall rules. This strategy can also involve introducing host-based endpointinspection agents that don't change the forwarding architecture of your VPC networkthrough multiple network interface VMs.

For an additional example of this configuration, see theStateful L7 firewall between VPC networks reference architecture.

Use fewer, broader firewall rule sets when possible

Only a certain number of rules can be programmed on anyVM. However, you can combine many rules into one complex ruledefinition. For example, if all VMs in the VPC network need to explicitly allow 10ingress TCP ports, you have two options: write 10 separate rules, each defininga single port, or define a single rule that includes all 10 ports. Defining asingle rule that includes all 10 ports is the more efficient option.

Create a generic rule set that applies to the entire VPC network, and then use morespecific rule sets for smaller groupings of VMs usingtargets.In other words, start by defining broad rules, and progressively define rulesmore narrowly as needed:

1. Apply firewall rules that are common across all VMs in the VPC network.
2. Apply firewall rules that can be grouped across several VMs, like aservice instance group or subnet.
3. Apply firewall rules to individual VMs, such as a NAT gateway or bastionhost.

Isolate VMs using service accounts when possible

Many organizations have environments that require specific rules for asubset of the VMs in a VPC network. There are two common approaches that you can take inthese cases: subnet isolation and target filtering.

Subnet isolation

With subnet isolation, thesubnet forms the security boundary across which Google Cloud firewall rules are applied.This approach is common in on-premises networking constructs and in cases whereIP addresses and network placement form part of the VM identity.

You can identify the VMs on a specific subnet byapplying a uniqueTag, network tag, orservice account to those instances. This lets you create firewall rules thatonly apply to the VMs in a subnet—those withthe associated Tag, network tag, service account. For example, to create a firewallrule that permits all communication between VMs in the same subnet, you can usethe following rule configuration on theFirewall rules page:

• Targets:Specified target tags
• Target tags:subnet-1
• Source filter:Subnets
• Subnets: Select subnet by name (example:subnet-1).

Target filtering

With target filtering, all VMs either reside on the same subnet or are partof an arbitrary set of subnets. With this approach, subnet membership is notconsidered part of the instance identity for firewall rules. Instead, you canuse Tags, network tags, or service accounts to restrict access between VMs in the samesubnet. Each group of VMs that uses the same firewall rules has the same networktag applied.

To illustrate this, consider a three-tier (web, app, database) application forwhich all of the instances are deployed in the same subnet. The web tier cancommunicate with end users and the app tier, and the app tier can communicatewith the database tier, but no other communication between tiers is allowed. Theinstances running the web tier have a network tag ofweb, the instancesrunning the app tier have a network tag ofapp, and the instances running thedatabase tier have a network tag ofdb.

The following firewall rules implement this approach:

However, even though it is possible to use network tags for target filtering in thismanner, we recommend that you use Tags or service accounts where possible. Target tagsare not access-controlled and can be changed by someone with theinstanceAdminrole while VMs are in service. Tags and service accounts are access-controlled, meaningthat a specific user must be explicitly authorized to use a service account.There can only be one service account per instance, whereas there can bemultiple tags. Also, service accounts assigned to a VM can only be changed whenthe VM is stopped.

Use automation to monitor security policies when using tags

If you use network tags, remember that an instance administrator can change those tags.This can circumvent security policy. Therefore, if you do use network tags in aproduction environment, use an automation framework to help you overcome thelack of IAM governance over the network tags.

Use additional tools to help secure and protect your apps

In addition to firewall rules, use these additional tools to help secure andprotect your apps:

• Use a Google Cloud globalHTTP(S) load balancer to support high availability and protocol normalization.
• IntegrateGoogle Cloud Armor with the HTTP(S) load balancer to provide DDoS protection and the abilityto block or allow IP addresses at the network edge.
• Control access to apps by usingIAP (IAP) to verify user identity and the context of the requestto determine if a user should be granted access.
• Provide a single interface for security insights, anomaly detection, andvulnerability detection withSecurity Command Center.

Network services: NAT and DNS

Use fixed external IP addresses with Cloud NAT

If you need fixed external IP addresses from a range of VMs, useCloud NAT.An example of why you might need fixed outbound IP addresses is the case inwhich a third party allows requests from specific external IP addresses. UsingCloud NAT lets you have a small number of NAT IP addressesfor each region that are used for outbound communications.

Cloud NAT also allows your VM instances to communicate acrossthe internet without having their own external IP addresses. Google Cloud firewallrules are stateful. This means that if a connection is allowed between a sourceand a target or a target and a destination, then all subsequent traffic in eitherdirection will be allowed as long as the connection is active. In other words,firewall rules allow bidirectional communication after a session is established.

Reuse IP addresses across VPCs with Cloud NAT

IP addresses can be reused across VPCs when Cloud NAT forNCC is enabled. Inter-VPC Cloud NAT is available whenVPCs are connected using NCC VPC spokes. If the VPC IP addresses overlap with ranges in external networks, enableHybrid NAT. Only connections initiatedfrom workloads in Google Cloud towards external networks are translated.

Use private DNS zones for name resolution

Useprivate zones on Cloud DNS to allow your services to be resolved with DNS within your VPC network using theirinternal IP addresses without exposing this mapping to the outside.

Usesplit horizon DNS to map services to different IP addresses from within the VPC network thanfrom the outside. For example, you can have a service exposed through networkload balancing from the public internet, but have internal load balancingprovide the same service using the same DNS name from within the VPC network.

API access for Google managed services

Use the default internet gateway where possible

Access from resources within the VPC network to Google APIs follows the defaultinternet gateway next-hop. Despite the next-hop gateway's name, the traffic pathfrom instances to the Google APIs remains within Google's network.

By default, only VM instances with an external IP address can communicate withGoogle APIs and services. If you need access from instances without an externalIP address, set up Private Service Connect endpoints or use thePrivate Google Access feature for each subnet. Thisdoesn't slow down communications for Google APIs.

Google Kubernetes Engine (GKE) automatically enablesPrivate Google Access on subnets where nodes are deployed. All nodes onthese subnets without an external IP address are able to access Google managedservices.

Add explicit routes for Google APIs if you need to modify the default route

If you need to modify the default route, then add explicit routes for GoogleAPI destination IP ranges.

In environments where the default route (0.0.0.0/0) doesn't use the defaultinternet gateway next-hop, configure explicit routes for thedestination IP address ranges used by Google APIs. Set the next-hop of the explicit routes to the defaultinternet gateway. An example of such a scenario is when you need to inspect alltraffic through an on-premises device.

Deploy instances that use Google APIs on the same subnet

Deploy instances that require access to Google APIs and services on the samesubnet and enable Private Google Access for instances without externalIP addresses. Alternatively, set up Private Service Connect endpoints.

If you are accessing Google APIs from your on-premises environment usingPrivate Google Access, useConfiguring Private Google Access for on-premises hosts to access some Google services over private IP address ranges. Check to seewhichservices are supported before activating this feature, because access to other Google APIs throughthe IP addresses provided by this service will be unreachable. Activating thisfeature can require additional DNS configuration, such as configuring DNSViews.

If you are accessing Google APIs from your on-premises environment usingPrivate Service Connect endpoints, seeAccess the endpoint fromon-premises hostsfor details.

Logging, monitoring, and visibility

Tailor logging for specific use cases and intended audiences

UseVPC Flow Logs for network monitoring, forensics, real-time security analysis, and expenseoptimization. You can enable or disable VPC Flow Logs at the subnet level.If VPC Flow Logs are enabled for a subnet, it collects data from all VMinstances in that subnet.

These logs record a sample of network flows that VM instances send and receive.Your logging use cases help to determine which subnets you deciderequire logging, and for how long.

Flow logs are aggregated by connection at 5-second intervals fromCompute Engine VMs and then exported in real time. You can view flow logs inCloud Logging and export them to any destination that Cloud Logging exportsupports.

The logs ingestion page in Logging tracks the volume of logs inyour project and lets you disable all logs ingestion or exclude (discard) logentries you're not interested in, so that you can minimize any charges for logsover your monthly allotment.

Logs are a critical part of both operational and security success, but theyaren't useful unless you review them and take action. Tailor logs for theirintended audience, which helps to ensure operational and security success foryour VPC networks.

For details, seeUsing VPC Flow Logs.

Increase log aggregation interval for VPC networks with long connections

For VPC networks with mostly long-lived connections, set the log aggregation intervalto 15 minutes to greatly reduce the number of logs generated and to enablequicker and simpler analysis.

An example workflow for which increasing the log aggregation interval isappropriate is network monitoring, which involves the following tasks:

• Performing network diagnosis
• Filtering the flow logs by VMs and by applications to understand trafficchanges
• Analyzing traffic growth to forecast capacity

Use VPC Flow Logs sampling to reduce volume

Use VPC Flow Logs sampling to reduce the volume of VPC Flow Logs, but still beable to see low-level samples and aggregated views.

An example workflow for which using sampling to reduce volume is appropriate isunderstanding network usage and optimizing network traffic expense. This workflowinvolves the following tasks:

• Estimating traffic between regions and zones
• Estimating traffic to specific countries on the internet
• Identifying top talkers

Remove additional metadata when you only need IP and port data

In security use cases where you are only interested in IP addresses and ports,remove the additional metadata to reduce the volume of data consumed inCloud Logging.

An example workflow for which removing metadata is appropriate is networkforensics, which involves the following tasks:

• Determining which IPs talked with whom and when
• Identifying any compromised IP addresses, found by analyzing network flows

Use Network Intelligence Center to get insights into your networks

Network Intelligence Center provides a singleconsole for managing Google Cloud network visibility, monitoring, andtroubleshooting. The following sections provide details about the Network Intelligence Centercomponents.

Network Topology

UseNetwork Topologyto visualize your network topology.

Connectivity Tests

UseConnectivity Teststo help diagnose connectivity issues with your VPC networks.

Performance Dashboard

UsePerformance Dashboardto check on the performance of the physical networking underlying yourVPC virtual networks.

Firewall Insights

UseFirewall Insightsto gain understandings about your firewall rules and how they interact.

Network Analyzer

UseNetwork Analyzerto monitor your VPC network configurations and to detectmisconfigurations and suboptimal configurations.

Flow Analyzer

UseFlow Analyzerto gain a better understanding of VPC traffic flows.

Reference architectures

This section highlights a few architectures that illustrate some of the bestpractices in this document.

Single project, single VPC network

This initial reference architecture includes all of the components necessary todeploy highly available architectures across multiple regions, with subnet-levelisolation and a 99.99% SLA connecting to your on-premises data centers.

Single host project, multiple service projects, single Shared VPC

Building on the initial reference architecture, Shared VPC host projects andmultiple service projects let administrators delegate administrativeresponsibilities—such as creating and managing instances—to Service ProjectAdmins while maintaining centralized control over network resources likesubnets, routes, and firewalls.

Multiple host projects, multiple service projects, multiple Shared VPC

The following diagram illustrates an architecture for VPC isolation, whichbuilds on our high-availability design while separating prod from otherprojects. There are many reasons to consider VPC isolation, including audit requirements (such asPCI), quota considerations between environments, or just another layer oflogical isolation. You only require two interconnects (for redundancy)per location but can add multiple Interconnect attachments to multiple VPC networks orregions from those.

Using isolation can also introduce the need for replication, as you decide where toplace core services such as proxies, authentication, and directory services.Using a Shared Services VPC network can help to avoid this replication, and allowyou to share these services with other VPC networks through NCC, while at thesame time centralizing administration and deployment.

Stateful L7 firewall between VPC networks

Thisarchitecture has multiple VPC networks that are bridged by an L7 next-generation firewall (NGFW) appliance, whichfunctions as a multi-NIC bridge between VPC networks.

An untrusted, outside VPC network is introduced to terminatehybrid interconnects and internet-based connections that terminate on theoutside leg of the L7 NGFW for inspection. There are many variations on thisdesign, but the key principle is to filter traffic through the firewall beforethe traffic reaches trusted VPC networks.

This design requires each VPC network to reside in the project where you insert theVM-based NGFW. Because quotas are enforced at the project level, you mustconsider the aggregate of all VPC resources.

Multiple VPC networks interconnected with NCC

This architecture has multiple VPC networks that connect to each other usingNCC. A transit VPC network is introduced to terminate hybridinterconnects and share the hybrid connectivity across all otherVPCs, which avoids the need to create VLAN attachments for eachVPC network. This approach consolidates theexternal connectivity and its associated routing considerations. Similarly, oneor more shared services VPC networks can be introduced to host common servicessuch as proxies, authentication, and directory services. There are manyvariations on this design, but the key principle is to handle the differentservices and connection types as spokes to a NCC hub thatprovides any-to-any connectivity amongst these. This reference architecture isdescribed in detail inCross-Cloud Network inter-VPC connectivity using Network Connectivity Center.

What's next

• Cross-Cloud Network for distributed applications
• VPC deep dive and best practices (Cloud NEXT'18 video)
• Hybrid and multi-cloud network topologies
• Decide a resource hierarchy for your Google Cloud landing zone
• Best practices for Compute Engine region selection
• Google Cloud for data center professionals: networking
• VPC documentation
• GKE networking overview
• Explore reference architectures, diagrams, and best practices about Google Cloud.Take a look at ourCloud Architecture Center.