Create an internal load balancer across VPC networks

This page explains how to create aninternal passthrough Network Load Balancer onGoogle Kubernetes Engine (GKE) across VPC networks.

Before reading this page, ensure that you'refamiliar with the following concepts:

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Create an internal load balancer with Private Service Connect

As a service producer, you can use service attachments to make your servicesavailable to service consumers in other VPC networks usingPrivate Service Connect. You cancreate, manage, and delete service attachments using aServiceAttachmentcustom resource.

Requirements and limitations

  • Limitations forPrivate Service Connectapply.
  • You can create a service attachment in GKE versions1.21.4-gke.300 and later.
  • Your cluster must have theHttpLoadBalancingadd-onenabled. NewGKE clusters have theHttpLoadBalancing add-on enabled bydefault.
  • You cannot use the same subnet in multiple service attachment configurations.
  • You must create a GKE service that uses an internal passthrough Network Load Balancer.
  • You cannot specify a subnet in a different project (Shared VPC) forGKE versions earlier than 1.22.4-gke.100. For Shared VPC,ensure allrequirements for Shared VPCare met.
  • After you create a service attachment, you can't update the internal loadbalancer. To change your load balancer configurations, you must delete andrecreate the service attachment.

Create aServiceAttachment

  1. Create a subnet.

    You must create anew subnetfor eachServiceAttachment.

    gcloudbetacomputenetworkssubnetscreateSUBNET_NAME\--projectPROJECT_ID\--networkNETWORK_NAME\--regionREGION\--rangeSUBNET_RANGE\--purposePRIVATE_SERVICE_CONNECT

    Replace the following:

    • SUBNET_NAME: the name of the new subnet. InGKE versions 1.22.4-gke.100 and later, you can specifya subnet in a different project by using the fully qualified resourceURL for this field. You can get the fully qualified resource URL usingthe commandgcloud compute networks subnets describe.
    • PROJECT_ID: the ID of your Google Cloudproject.
    • NETWORK_NAME: the name of the VPCnetwork for the subnet.
    • REGION: the region for the new subnet. You must usethe same region as the service that you create.
    • SUBNET_RANGE: the IP address range to use for thesubnet.

  2. Deploy a workload.

    The following manifest describes a Deployment that runs a sample webapplication container image. Save the manifest asmy-deployment.yaml:

    apiVersion:apps/v1kind:Deploymentmetadata:name:psc-ilbspec:replicas:3selector:matchLabels:app:psc-ilbtemplate:metadata:labels:app:psc-ilbspec:containers:-name:whereamiimage:us-docker.pkg.dev/google-samples/containers/gke/whereami:v1ports:-name:httpcontainerPort:8080readinessProbe:httpGet:path:/healthzport:8080scheme:HTTPinitialDelaySeconds:5timeoutSeconds:1

  3. Apply the manifest to your cluster:

    kubectlapply-fmy-deployment.yaml

  4. Create a service.The following manifest describes a service that creates an internal passthrough Network Load Balanceron TCP port 8080. Save the manifest asmy-service.yaml:

    apiVersion:v1kind:Servicemetadata:name:SERVICE_NAMEannotations:networking.gke.io/load-balancer-type:"Internal"spec:type:LoadBalancerselector:app:psc-ilbports:-port:80targetPort:8080protocol:TCP

    Replace the following:

    • SERVICE_NAME: the name of the new service.

  5. Apply the manifest to your cluster:

    kubectlapply-fmy-service.yaml

  6. CreateServiceAttachment.

    The following manifest describes aServiceAttachment that exposes theservice that you created to service consumers. Save the manifest asmy-psc.yaml:

    apiVersion:networking.gke.io/v1kind:ServiceAttachmentmetadata:name:SERVICE_ATTACHMENT_NAMEnamespace:defaultspec:connectionPreference:ACCEPT_AUTOMATICnatSubnets:-SUBNET_NAMEproxyProtocol:falseresourceRef:kind:Servicename:SERVICE_NAME
    Note: For GKE versions earlier than 1.23.3-gke.900, usenetworking.gke.io/v1beta1 instead ofnetworking.gke.io/v1.

    Replace the following:

    Note: TheServiceAttachment resource will have error events until theGKE controller creates the internal passthrough Network Load Balancer that is definedby the service. After the internal passthrough Network Load Balancer is provisioned, the errors areno longer published.

    For more information about the manifest fields, see theservice attachment fields.

  7. Apply the manifest to your cluster:

    kubectlapply-fmy-psc.yaml

  8. Verify that the Private Service Connect controller created theservice attachment:

    gcloudbetacomputeservice-attachmentslist

    The output shows a service attachment with an automatically generated name:

    NAME        REGION       PRODUCER_FORWARDING_RULE          CONNECTION_PREFERENCEk8s1-sa-... REGION_NAME  a3fea439c870148bdba5e59c9ea9451a  ACCEPT_AUTOMATIC

View aServiceAttachment

You can view the details of aServiceAttachment using the following command:

kubectldescribeserviceattachmentSERVICE_ATTACHMENT_NAME

The output is similar to the following:

Name:        <sa-name>Namespace:    defaultLabels:       <none>Annotations:  <none>API Version:  networking.gke.io/v1beta1Kind:         ServiceAttachmentMetadata:  ...Status:  Forwarding Rule URL:      https://www.googleapis.com/compute/beta/projects/<project>/regions/<region>/forwardingRules/<fr-name>  Last Modified Timestamp:  2021-07-08T01:32:39Z  Service Attachment URL:   https://www.googleapis.com/compute/beta/projects/<projects>/regions/<region>/serviceAttachments/<gce-service-attachment-name>Events:                     <none>

Consume aServiceAttachment

To consume your service from another project, perform the following steps:

  1. Get the URL of theServiceAttachment:

    kubectlgetserviceattachmentSERVICE_ATTACHMENT_NAME-o=jsonpath="{.status.serviceAttachmentURL}"

    The output is similar to the following:

      serviceAttachmentURL: https://www.googleapis.com/compute/alpha/projects/<project>/region/<region>/serviceAttachments/k8s1-...my-sa

  2. Create a Private Service Connect endpointusing the URL of theServiceAttachment.

  3. Verify that you can connect to the Service that you deployed in the producerproject by using acurl command from a VM in the consumer project:

    curlPSC_IP_ADDRESS

    ReplacePSC_IP_ADDRESS with the IP address of theforwarding rule in the consumer project.

    The output is similar to the following:

    {  "cluster_name":"cluster",  "host_header":"10.128.15.200",  "node_name":"gke-psc-default-pool-be9b6e0e-dvxg.c.gke_project.internal",  "pod_name":"foo-7bf648dcfd-l5jf8",  "pod_name_emoji":"👚",  "project_id":"gke_project",  "timestamp":"2021-06-29T21:32:03",  "zone":"ZONE_NAME"}

Update aServiceAttachment

You can update aServiceAttachment using the following steps:

Note: You cannot update theresourceRef field and you can only add subnets tothenatSubnets field. You cannot remove subnets. If you need to update theresourceRef field or remove subnets from thenatSubnets field, you mustdelete and then recreate theServiceAttachment.

  1. Edit theServiceAttachment manifest inmy-psc.yaml:

    apiVersion:networking.gke.io/v1kind:ServiceAttachmentmetadata:name:my-sanamespace:defaultspec:connectionPreference:ACCEPT_AUTOMATICnatSubnets:-my-nat-subnetproxyProtocol:falseresourceRef:kind:Servicename:ilb-service
    Note: For GKE versions earlier than 1.23.3-gke.900, usenetworking.gke.io/v1beta1 instead ofnetworking.gke.io/v1.

  2. Apply the manifest to your cluster:

    kubectlapply-fmy-psc.yaml

Delete aServiceAttachment

You cannot delete an internal passthrough Network Load Balancer that is connected to a serviceattachment. You must delete the service attachment and GKEService separately.

  1. Delete the service attachment:

    kubectldeleteserviceattachmentSERVICE_ATTACHMENT_NAME--wait=false

    This command marks the service attachment for deletion, but the resourcecontinues to exist. You can also wait for the deletion to finish by omittingthe--wait flag.

  2. Delete the Service:

    kubectldeletesvcSERVICE_NAME

  3. Delete the subnet:

    gcloudcomputenetworkssubnetsdeleteSUBNET_NAME

ServiceAttachment fields

TheServiceAttachment has the following fields:

Troubleshooting

You can view error messages using the following command:

kubectlgetevents-nNAMESPACE

ReplaceNAMESPACE with the namespace of theinternal passthrough Network Load Balancer.

Error deleting internal passthrough Network Load Balancer

An error message similar to the following occurs if you try to delete aninternal passthrough Network Load Balancer that is being used by a service attachment. You must deletetheServiceAttachment before you can delete the internal passthrough Network Load Balancer.

Error syncing load balancer: failed to ensure load balancer: googleapi:Error 400: The forwarding_rule resource '<fwd-rule-URL>' is already being usedby '<svc-attachment-URL>', resourceInUseByAnotherResource.

Error enabling global access

You might encounter an error when you enableglobalaccessfor an internal passthrough Network Load Balancer that uses a Private Service Connect serviceattachment.

Symptom:

When you enable global access by updating your GKE Servicemanifest and setting thenetworking.gke.io/internal-load-balancer-allow-global-access: "true"annotation, the update fails when you apply the manifest. You see thefollowing error message when you run thekubectl get events command:

Error syncing load balancer: failed to ensure load balancer: googleapi: Error 400: The forwarding_rule resource '...' is already being used by '...', resourceInUseByAnotherResource

Reason:

This error occurs because the GKE control plane attempts tore-create the load balancer's forwarding rule to enable global access. However,because the forwarding rule is in use by aServiceAttachment, the rule cannotbe deleted and re-created, which results in the error.

Workaround:

To enable global access on the load balancer while minimizing downtime, youmust update the GKE Service configurationand manually updatethe forwarding rule:

  1. Update the GKE Service manifest: update yourGKE Service manifest to include thenetworking.gke.io/internal-load-balancer-allow-global-access: "true"annotation.

  2. Manually enable global access on the forwarding rule: follow the steps inEnable globalaccessto update the load balancer's forwarding rule by using either theGoogle Cloud console, theGoogle Cloud CLI, or the Compute Engine API.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.