This page shows you how to create a Kubernetes Service that is backed by azonalGCE_VM_IP_PORT network endpoint group(NEG) in a Google Kubernetes Engine (GKE)VPC-native cluster.

For information on the benefits, requirements, and limitations ofcontainer-native load balancing, seeContainer-native loadbalancing.

Overview

ANEG represents a groupof endpoints. GKE supports standalone NEGs of theGCE_VM_IP_PORT type.GCE_VM_IP_PORT NEGs support endpoints using either theVM's primary internal IP address or an IP address from one of its alias IPranges.

In the context of a GKE VPC-native clusterusing standalone NEGs, each endpoint is a Pod IP address and target port. PodIP addresses are sourced from the node's alias IP range for Pods, which comesfrom the cluster'ssubnet secondary IP address range forPods.

GKE provides aNEG controller to manage the membership ofGCE_VM_IP_PORT NEGs. You can add the NEGs it creates as backends to thebackend services for load balancers that you configure outside of theGKE API.

The following diagram describes how Kubernetes API objects correspond toCompute Engine objects.

Ingress with NEGs

WhenNEGs are used with GKE Ingress,the Ingress controller facilitates the creation of all aspects of the loadbalancer. This includes creating the virtual IP address, forwarding rules,health checks, firewall rules, and more.

Ingress is the recommended way to use container-native load balancing as it hasmany features that simplify the management of NEGs. Standalone NEGs are anoption if NEGs managed by Ingress don't serve your use case.

Standalone NEGs

When NEGs are deployed with load balancers provisioned by anything other thanIngress, they are considered standalone NEGs. Standalone NEGs are deployed andmanaged through the NEG controller, but the forwarding rules, health checks, andother load balancing objects are deployed manually.

Standalone NEGs don't conflict with Ingress enabled container-native loadbalancing.

The following illustration shows the differences in how the load balancingobjects are deployed in each scenario:

Preventing leaked NEGs

With standalone NEGs, you are responsible for managing the lifecycles of NEGsand the resources that make up the load balancer. You could leak NEGs in theseways:

• When a GKE service is deleted, the associated NEG won't begarbage collected if the NEG is still referenced by a backend service.Dereference the NEG from the backend service to allow NEG deletion.

• When a cluster is deleted, standalone NEGs won't be deleted in the followingscenarios:

  • The NEG is still referenced by a backend service.
  • The cluster deletion process shuts down the NEG controller before thecontroller can delete the NEG.

  To prevent leaked NEGs, dereference the NEG from the backend service anddelete all NEGs before you delete the cluster.

If you have leaked NEGs after deleting the cluster or service, you candelete the NEGs by using the Google Cloud CLI.

Use cases of standalone NEGs

Standalone NEGs have several critical uses. Standalone NEGs are very flexible.This is in contrast to Ingress (used with or without NEGs) which defines aspecific set of load balancing objects that were chosen in an opinionated way tomake them straightforward to use.

Use cases for standalone NEGs include:

Heterogeneous services of containers and VMs

NEGs can contain both VM and container IP addresses. This means a single virtualIP address can point to a backend that consists of both Kubernetes andnon-Kubernetes workloads. This can also be used to migrate existing workloadsto a GKE cluster.

Standalone NEGs can point to VM IPs which makes it possible to manuallyconfigure load balancers to point at backends that are comprised of both VMs andcontainers for the same service VIP.

Customized Ingress controllers

You can use a customized Ingress controller (or no Ingress controller) toconfigure load balancers that target standalone NEGs.

Use Cloud Service Mesh with GKE

You can useCloud Service Mesh with GKE.Cloud Service Mesh uses standalone NEGs to provide container-native load balancing forthe managed service mesh.

Use external proxy Network Load Balancers with GKE

You can use standalone NEGs to load balance directly to containers with theexternal proxy Network Load Balancer which is not supportednatively by Kubernetes/GKE.

Pod readiness

Readinessgates are an extensibility feature of Kubernetes that allow the injection of extra feedback or signals into the PodStatus to allow the Pod to transition to theReady state. The NEG controller manages a custom readiness gate to ensure thefull network path from Compute Engine load balancer to pod is functional. Pod readinessgates in GKE are explained incontainer-native loadbalancing.

Ingress with NEGs deploys and manages Compute Engine health checks on behalfof the load balancer. However, standalone NEGs make no assumptions aboutCompute Engine health checks because they are expected to be deployed andmanaged separately. Compute Engine health checks should always be configuredalong with the load balancer to prevent traffic from being sent to backendswhich are not ready to receive. If there is no health check status associatedwith the NEG (usually because no health check is configured), then the NEGcontroller will mark the Pod's readiness gate value to True when itscorresponding endpoint is programmed in NEG.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Ensure that you have an existing VPC-nativecluster. Your cluster must have theHttpLoadBalancing add-on enabled. New GKE clusters have theHttpLoadBalancing add-on enabled by default.

  To create a new Standard cluster, seeCreate a VPC-native cluster.Autopilot clusters are VPC-native by default.

Using standalone NEGs

The following instructions show you how to use standalone NEGs with an externalHTTP load balancer on GKE.

You must create the following objects:

• ADeployment that creates and manages Pods.
• AService that creates a NEG.
• A load balancer created with the Compute Engine API. This differs from using NEGswith Ingress, in which case Ingress creates and configures a load balancer foryou. With standalone NEGs you are responsible for associating the NEG and thebackend service to connect the Pods to the load balancer. The load balancerconsists of several components, shown in the following diagram:

Create a Deployment

The following example manifests specify Deployments that run three instances ofa containerized HTTP server. The HTTP server responds to requests with thehostname of the application server, the name of the Pod the server is running on.

We recommend you use workloads that use Pod readiness feedback.

apiVersion:apps/v1kind:Deploymentmetadata:labels:run:neg-demo-app# Label for the Deploymentname:neg-demo-app# Name of Deploymentspec:replicas:3selector:matchLabels:run:neg-demo-apptemplate:# Pod templatemetadata:labels:run:neg-demo-app# Labels Pods from this Deploymentspec:# Pod specification; each Pod created by this Deployment has this specificationcontainers:-image:registry.k8s.io/serve_hostname:v1.4# Application to run in Deployment's Podsname:hostname

apiVersion:apps/v1kind:Deploymentmetadata:labels:run:neg-demo-app# Label for the Deploymentname:neg-demo-app# Name of Deploymentspec:minReadySeconds:60# Number of seconds to wait after a Pod is created and its status is Readyreplicas:3selector:matchLabels:run:neg-demo-apptemplate:# Pod templatemetadata:labels:run:neg-demo-app# Labels Pods from this Deploymentspec:# Pod specification; each Pod created by this Deployment has this specificationcontainers:-image:registry.k8s.io/serve_hostname:v1.4# Application to run in Deployment's Podsname:hostname

Save this manifest asneg-demo-app.yaml, then create the Deployment by runningthe following command:

kubectlapply-fneg-demo-app.yaml

Create a Service

The following manifest specifies a Service where:

• Any Pod with the labelrun: neg-demo-app is a member of this Service.
• The Service has oneServicePortfield with port 80.
• Thecloud.google.com/neg annotation specifies that port 80 will beassociated with a NEG. The optionalname field specifies that the NEG willbe namedNEG_NAME. If thename field is omitted, a unique namewill be automatically generated. Seenaming NEGs for details.
• Each member Pod must have a container that is listening on TCP port 9376.

apiVersion:v1kind:Servicemetadata:name:neg-demo-svcannotations:cloud.google.com/neg:'{"exposed_ports":{"80":{"name":"NEG_NAME"}}}'spec:type:ClusterIPselector:run:neg-demo-app# Selects Pods labelled run: neg-demo-appports:-port:80protocol:TCPtargetPort:9376

ReplaceNEG_NAME with the name for the NEG. The NEG namemust be unique in its region.

Save this manifest asneg-demo-svc.yaml, then create the Service by runningthe following command:

kubectlapply-fneg-demo-svc.yaml

A NEG is created within a few minutes of Service creation.

Service types

While this example uses aClusterIP service, all fivetypes ofServicessupport standalone NEGs. We recommend the default type,ClusterIP.

Naming NEGs

In GKE versions 1.18.18-gke.1200 and later, you can specify acustom name for NEGs, or GKE can generate a name automatically.Previous versions of GKE only support automatically generatedNEG names.

GKE creates one NEG in each zone used by the cluster. TheNEGs all use the same name.

Specifying a name

Specifying a custom NEG name simplifiesconfiguring the loadbalancer because you know the name and zone(s) of the NEG(s) inadvance. Custom NEG names must meet the following requirements:

• Be unique to the cluster's zone for zonal clusters, or unique to the regionfor regional clusters.

• Must not match the name of any existing NEG that was not created by theGKE NEG controller.

• Must not contain underscores.

Use thename field in thecloud.google.com/neg annotation of the Service tospecify a NEG name:

cloud.google.com/neg:'{"exposed_ports":{"80":{"name":"NEG_NAME"}}}'

ReplaceNEG_NAME with the name for the NEG. The NEG namemust be unique in its region.

Using an automatically generated name

Automatically generated NEG names are guaranteed to be unique. To use anautomatically generated name, omit thename field:

cloud.google.com/neg:'{"exposed_ports":{"80":{}}}'

The automatically generated name has the following format:

k8s1-CLUSTER_UID-NAMESPACE-SERVICE-PORT-RANDOM_HASH

Mapping ports to multiple NEGs

A Service can listen on more than one port. By definition, NEGs have only asingle IP address and port. This means that if you specify a Service withmultiple ports, it will create a NEG for each port.

The format of thecloud.google.com/neg annotation is:

cloud.google.com/neg:'{"exposed_ports":{"SERVICE_PORT_1":{},"SERVICE_PORT_2":{},"SERVICE_PORT_3":{},...}}'

In this example, each instance ofSERVICE_PORT_N is adistinct port number that refers to existing service ports of the Service. Foreach service port listed, the NEG controller creates one NEG in each zone thecluster occupies.

Retrieve NEG statuses

Use the following command to retrieve the statuses of the cluster's Services:

kubectlgetserviceneg-demo-svc-oyaml

The output is similar to the following:

cloud.google.com/neg-status: '{   "network-endpoint-groups":{      "SERVICE_PORT_1": "NEG_NAME_1",      "SERVICE_PORT_2": "NEG_NAME_2",      ...   },   "zones":["ZONE_1", "ZONE_2", ...]}

In this output, each element in thenetwork-endpoint-groups mapping is a service port(likeSERVICE_PORT_1) and the name of the correspondingmanaged NEGs (likeNEG_NAME_1). Thezones listcontains every zone (likeZONE_1) that has a NEG in it.

The output is similar to the following:

apiVersion:v1kind:Servicemetadata:annotations:cloud.google.com/neg:'{"exposed_ports":{"80":{}}}'cloud.google.com/neg-status:'{"network_endpoint_groups":{"80":"k8s1-cca197ad-default-neg-demo-app-80-4db81e02"},"zones":["ZONE_1","ZONE_2"]}'labels:run:neg-demo-appname:neg-demo-appnamespace:defaultselfLink:/api/v1/namespaces/default/services/neg-demo-app...spec:clusterIP:10.0.14.252ports:-port:80protocol:TCPtargetPort:9376selector:run:neg-demo-appsessionAffinity:Nonestatus:loadBalancer:{}

In this example, the annotation shows that service port 80 is exposed to NEGsnamedk8s1-cca197ad-default-neg-demo-app-80-4db81e02.

Validate NEG creation

A NEG is created within a few minutes of Service creation. If there are Podsthat match the label specified in the Service manifest, then upon creation theNEG will contain the IPs of the Pods.

There are two ways to verify that the NEG is created and is correctlyconfigured. In GKE 1.18.6-gke.6400 and later, a custom resourceServiceNetworkEndpointGroup stores status information about NEGs created bythe Service controller. In previous versions, you must inspect the NEGsdirectly.

TheServiceNetworkEndpointGroup resource

List the NEGs in a cluster by getting all of theServiceNetworkEndpointGroup resources:

kubectlgetsvcneg

Observe the status of a NEG by checking the status of theServiceNetworkEndpointGroup resource:

kubectlgetsvcnegNEG_NAME-oyaml

ReplaceNEG_NAME with the name of the individual NEG youwant to inspect.

The output of this command includes a status section that might contain errormessages. Some errors are reported as a Service event. You can find furtherdetails by querying the Service object:

kubectldescribeserviceSERVICE_NAME

ReplaceSERVICE_NAME with the name of the relevant Service.

To verify that the NEG controller is successfully syncing the NEG, check thestatus field of theServiceNetworkEndpointGroup resource for a condition withtype:Synced. The time of the most recent sync is in thestatus.lastSyncTimefield.

ServiceNetworkEndpointGroup resources only exist in GKE version1.18 and later.

Inspecting NEGs directly

Verify that the NEG exists by listing the NEGs in your Google Cloudproject and checking for a NEG that matches the Service you created. The NEG'sname has the following format:

k8s1-CLUSTER_UID-NAMESPACE-SERVICE-PORT-RANDOM_HASH

Use the following command to list NEGs:

gcloudcomputenetwork-endpoint-groupslist

The output is similar to the following:

NAME                                          LOCATION       ENDPOINT_TYPE   SIZEk8s1-70aa83a6-default-my-service-80-c9710a6fZONE_NAME      GCE_VM_IP_PORT  3

This output shows that theSIZE of the NEG is 3, meaning that it has threeendpoints which correspond to the three Pods in the Deployment.

Identify the individual endpoints with the following command:

gcloudcomputenetwork-endpoint-groupslist-network-endpointsNEG_NAME

ReplaceNEG_NAME with the name of the NEG for which youwant to display the individual endpoints.

The output shows three endpoints, each of which has a Pod's IP address andport:

INSTANCE                                           IP_ADDRESS  PORTgke-cluster-3-default-pool-4cc71a15-qlpf  10.12.1.43  9376gke-cluster-3-default-pool-4cc71a15-qlpf  10.12.1.44  9376gke-cluster-3-default-pool-4cc71a15-w9nk  10.12.2.26  9376

Attaching an external Application Load Balancer to standalone NEGs

You can use NEGs as a backend for an external Application Load Balancer using the Compute Engine API.

1. Create a firewall rule. Load balancers need to access clusterendpoints to perform health checks. This command creates a firewall rule toallow access:

   gcloudcomputefirewall-rulescreatefw-allow-health-check-and-proxy\--network=NETWORK_NAME\--action=allow\--direction=ingress\--target-tags=GKE_NODE_NETWORK_TAGS\--source-ranges=130.211.0.0/22,35.191.0.0/16\--rules=tcp:9376

   Replace the following:

   • NETWORK_NAME: the network wherethe cluster runs.
   • GKE_NODE_NETWORK_TAGS: thenetworking tagson the GKE nodes.

   If you did not create custom network tags for your nodes, GKEautomatically generates tags for you. You can look up these generated tags byrunning the following command:

   gcloudcomputeinstancesdescribeINSTANCE_NAME

   ReplaceINSTANCE_NAME with the name of the hostCompute Engine VM instance running the GKE node. For example,the output in the previoussection displays theinstance names in theINSTANCE column for the GKE nodes.

   For Standard clusters, you can also rungcloud compute instances listto list all instances in your project.

2. Create a global virtual IP address for the load balancer:

   gcloudcomputeaddressescreatehostname-server-vip\--ip-version=IPV4\--global

3. Create a health check. This is used by the load balancer to detect theliveness of individual endpoints within the NEG.

   gcloudcomputehealth-checkscreatehttphttp-basic-check\--use-serving-port

4. Create abackend service thatspecifies that this is a global external Application Load Balancer:

   gcloudcomputebackend-servicescreatemy-bes\--protocolHTTP\--health-checkshttp-basic-check\--global

5. Create aURL map andtargetproxy for the load balancer. Thisexample is very straightforward because theserve_hostname app used for this guidehas a single endpoint and does not feature URLs.

   gcloudcomputeurl-mapscreateweb-map\--default-servicemy-bes

   gcloudcomputetarget-http-proxiescreatehttp-lb-proxy\--url-mapweb-map

6. Create aforwarding rule.This is what creates the load balancer.

   gcloudcomputeforwarding-rulescreatehttp-forwarding-rule\--address=HOSTNAME_SERVER_VIP\--global\--target-http-proxy=http-lb-proxy\--ports=80

   ReplaceHOSTNAME_SERVER_VIP with the IP address to usefor the load balancer. If you omit--address, GKEautomatically assigns an ephemeral IP address.

   You can alsoreserve a new static external IP address.

Checkpoint

These are the resources you have created so far:

• An external virtual IP address
• The forwarding rules
• The firewall rules
• The target HTTP proxy
• The URL map the Compute Engine health check
• The backend service
• The Compute Engine health check

The relationship between these resources is shown in the following diagram:

These resources together are a load balancer. In the next step you will addbackends to the load balancer.

One of the benefits of standalone NEGs demonstrated here is that the lifecyclesof the load balancer and backend can be completely independent. The loadbalancer can continue running after the application, its services, or theGKE cluster is deleted. You can add and remove new NEGs ormultiple NEGs from the load balancer without changing any of the frontend loadbalancer objects.

Add backends to the load balancer

Usegcloud compute backend-servicesadd-backend toconnect the NEG to the load balancer by adding it as a backend of themy-bes backend service:

gcloudcomputebackend-servicesadd-backendmy-bes\--global\--network-endpoint-group=NEG_NAME\--network-endpoint-group-zone=NEG_ZONE\--balancing-modeRATE--max-rate-per-endpoint5

Replace the following:

• NEG_NAME: the name of your network endpointgroup. The name is either thename you specified when creating theNEG or an autogenerated name. If you did not specify a namefor the NEG, see the following instructions to find the autogenerated name.
• NEG_ZONE: the zone your networkendpoint group is in. See the following instructions to find this value.

Use this command to get the name and location of the NEG:

gcloudcomputenetwork-endpoint-groupslist

The output is similar to the following:

NAME                                          LOCATION       ENDPOINT_TYPE   SIZEk8s1-70aa83a6-default-my-service-80-c9710a6fZONE_NAME      GCE_VM_IP_PORT  3

In this example output, the name of the NEG isk8s1-70aa83a6-default-my-service-80-c9710a6f.

Multiple NEGs can be added to the same backend service. Global backend serviceslikemy-bes can have NEG backends in differentregions, while regional backend services must have backends in a single region.

Validate that the load balancer works

gcloudcomputebackend-servicesdescribemy-bes--global

backends:- balancingMode: RATE  capacityScaler: 1.0  group: ... /networkEndpointGroups/k8s1-70aa83a6-default-my-service-80-c9710a6f...healthChecks:- ... /healthChecks/http-basic-check...name: my-bes...

gcloudcomputebackend-servicesget-healthmy-bes--global

status:  healthStatus:  - healthState: HEALTHY    instance: ... gke-cluster-3-default-pool-4cc71a15-qlpf    ipAddress: 10.12.1.43    port: 50000  - healthState: HEALTHY    instance: ... gke-cluster-3-default-pool-4cc71a15-qlpf    ipAddress: 10.12.1.44    port: 50000  - healthState: HEALTHY    instance: ... gke-cluster-3-default-pool-4cc71a15-w9nk    ipAddress: 10.12.2.26    port: 50000

gcloudcomputeaddressesdescribehostname-server-vip--global|grep"address:"

curl34.98.102.37

Theproxy-only subnet isfor all regional internal Application Load Balancers in the load balancer's region.

gcloudcomputenetworkssubnetscreateproxy-only-subnet\--purpose=REGIONAL_MANAGED_PROXY\--role=ACTIVE\--region=COMPUTE_REGION\--network=lb-network\--range=10.129.0.0/23

POST https://compute.googleapis.com/compute/projects/PROJECT_ID/regions/COMPUTE_REGION/subnetworks

{"name":"proxy-only-subnet","ipCidrRange":"10.129.0.0/23","network":"projects/PROJECT_ID/global/networks/lb-network","region":"projects/PROJECT_ID/regions/COMPUTE_REGION","purpose":"REGIONAL_MANAGED_PROXY","role":"ACTIVE"}

Configuring firewall rules

This example uses the following firewall rules:

• fw-allow-ssh: An ingress rule, applicable to the instances being loadbalanced, that allows incoming SSH connectivity on TCP port 22 from anyaddress. You can choose a more restrictive source IP range for this rule. Forexample, you can specify just the IP ranges of the system from which youinitiate SSH sessions. This example uses the target tagallow-ssh toidentify the VMs to which the firewall rule applies.

• fw-allow-health-check: An ingress rule, applicable to the instances beingload balanced, that allows all TCP traffic from the Google Cloudhealth checking systems (in130.211.0.0/22 and35.191.0.0/16). Thisexample uses the target tagload-balanced-backend to identify the instancesto which it should apply.

• fw-allow-proxies: An ingress rule, applicable to the instances beingload balanced, that allows TCP traffic on port9376 from the internalHTTP(S) load balancer's managed proxies. This example uses thetarget tagload-balanced-backend to identify the instances to which itshould apply.

Without these firewall rules, thedefault denyingress rule blocks incomingtraffic to the backend instances.

POSThttps://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{"name":"fw-allow-ssh","network":"projects/PROJECT_ID/global/networks/lb-network","sourceRanges":["0.0.0.0/0"],"targetTags":["allow-ssh"],"allowed":[{"IPProtocol":"tcp","ports":["22"]}],"direction":"INGRESS"}

POSThttps://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{"name":"fw-allow-health-check","network":"projects/PROJECT_ID/global/networks/lb-network","sourceRanges":["130.211.0.0/22","35.191.0.0/16"],"targetTags":["load-balanced-backend"],"allowed":[{"IPProtocol":"tcp"}],"direction":"INGRESS"}

POSThttps://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{"name":"fw-allow-proxies","network":"projects/PROJECT_ID/global/networks/lb-network","sourceRanges":["10.129.0.0/23"],"targetTags":["load-balanced-backend"],"allowed":[{"IPProtocol":"tcp","ports":["9376"]}],"direction":"INGRESS"}

Configuring the load balancer

For the forwarding rule's IP address, use a backend subnet. If you try to usetheproxy-only subnet, forwarding rule creation fails.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/COMPUTE_REGION/healthChecks

{"name":"l7-ilb-gke-basic-check","type":"HTTP","httpHealthCheck":{"portSpecification":"USE_SERVING_PORT"}}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/COMPUTE_REGION/backendServices

{"name":"l7-ilb-gke-backend-service","backends":[{"group":"https://www.googleapis.com/compute/v1/projects/PROJECT_ID/zones/COMPUTE_ZONE/networkEndpointGroups/NEG_NAME","balancingMode":"RATE","maxRatePerEndpoint":5}],"healthChecks":["projects/PROJECT_ID/regions/COMPUTE_REGION/healthChecks/l7-ilb-gke-basic-check"],"loadBalancingScheme":"INTERNAL_MANAGED"}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/COMPUTE_REGION/urlMaps

{"name":"l7-ilb-gke-map","defaultService":"projects/PROJECT_ID/regions/COMPUTE_REGION/backendServices/l7-ilb-gke-backend-service"}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/COMPUTE_REGION/targetHttpProxy

{"name":"l7-ilb-gke-proxy","urlMap":"projects/PROJECT_ID/global/urlMaps/l7-ilb-gke-map","region":"COMPUTE_REGION"}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/COMPUTE_REGION/forwardingRules

{"name":"l7-ilb-gke-forwarding-rule","IPAddress":"10.1.2.199","IPProtocol":"TCP","portRange":"80-80","target":"projects/PROJECT_ID/regions/COMPUTE_REGION/targetHttpProxies/l7-ilb-gke-proxy","loadBalancingScheme":"INTERNAL_MANAGED","subnetwork":"projects/PROJECT_ID/regions/COMPUTE_REGION/subnetworks/backend-subnet","network":"projects/PROJECT_ID/global/networks/lb-network","networkTier":"PREMIUM",}

Testing

Create a VM instance in the zone to test connectivity:

gcloudcomputeinstancescreatel7-ilb-client\--image-family=debian-9\--image-project=debian-cloud\--zone=COMPUTE_ZONE\--network=lb-network\--subnet=backend-subnet\--tags=l7-ilb-client,allow-ssh

Sign in to the client instance to check that HTTP(S) services on the backends arereachable using the internal Application Load Balancer's forwarding rule IP address,and traffic is being load balanced among endpoints in the NEG.

Connect to each client instance using SSH:

gcloudcomputesshl7-ilb-client\--zone=COMPUTE_ZONE-b

Verify that the IP is serving its hostname:

curl10.1.2.199

For HTTPS testing, run the following command:

curl-k-s'https://test.example.com:443'--connect-totest.example.com:443:10.1.2.199:443

The-k flag causescurl to skip certificate validation.

Run 100 requests and confirm that they are load balanced.

For HTTP:

{RESULTS=foriin{1..100}doRESULTS="$RESULTS:$(curl--silent10.1.2.199)"doneecho"***"echo"*** Results of load-balancing to 10.1.2.199: "echo"***"echo"$RESULTS"|tr':''\n'|grep-Ev"^$"|sort|uniq-cecho}

For HTTPS:

{RESULTS=foriin{1..100}doRESULTS="$RESULTS:$(curl-k-s'https://test.example.com:443'--connect-totest.example.com:443:10.1.2.199:443)"doneecho"***"echo"*** Results of load-balancing to 10.1.2.199: "echo"***"echo"$RESULTS"|tr':''\n'|grep-Ev"^$"|sort|uniq-cecho}

Attach an external proxy Network Load Balancer to standalone NEGs

You can use standalone NEGs to load balance directly to containers with theexternal proxy Network Load Balancer which is not supportednatively by Kubernetes or GKE. Proxy Network Load Balancers areintended for TCP traffic only, with or without SSL. For HTTP(S) traffic, werecommend that you use an Application Load Balancer instead.

1. Create a firewall rule to allow health checks.

   Load balancers need to access cluster endpoints to perform health checks. Thefollowing command creates a firewall rule to allow access:

   gcloudcomputefirewall-rulescreateallow-tcp-lb-and-health\--network=NETWORK_NAME\--target-tags=GKE_NODE_NETWORK_TAGS\--source-ranges=130.211.0.0/22,35.191.0.0/16\--allowtcp:9376

   Replace the following:

   • NETWORK_NAME: the network wherethe cluster runs.
   • GKE_NODE_NETWORK_TAGS: thenetworking tagson the GKE nodes.

   If you didn't create custom network tags for your nodes, GKEautomatically generates tags for you. You can look up these generated tags byrunning the following command:

   gcloudcomputeinstancesdescribeINSTANCE_NAME

   ReplaceINSTANCE_NAME with the name of the hostCompute Engine VM instance running the GKE node. For example,the output in theInspect NEGs directly sectiondisplays the instance names in theINSTANCE column for theGKE nodes.

   For Standard clusters, you can also run thegcloud compute instances listto list all instances in your project.

   For Autopilot clusters, target the cluster service account insteadof network tags.

   Replace the--target-tags=GKE_NODE_NETWORK_TAGS flag with--target-service-accounts=SERVICE_ACCOUNT_EMAIL.We recommended that you use a customminimally-privilegedservice account for the cluster.

2. Create a global virtual IP address for the load balancer:

   gcloudcomputeaddressescreatetcp-lb-static-ipv4\--ip-version=IPV4\--global

3. Define a health check for backend endpoints.

   The load balancer uses health checks to detect theliveness of individual endpoints within the NEG.

   gcloudcomputehealth-checkscreatetcpmy-tcp-health-check\--use-serving-port

4. Create a backend service with session affinity.

   Thisbackend service specifies thatthis is a external proxy Network Load Balancer withCLIENT_IPsession affinity:

   gcloudcomputebackend-servicescreatemy-tcp-lb\--load-balancing-schemeEXTERNAL_MANAGED\--global-health-checks\--global\--protocolTCP\--health-checksmy-tcp-health-check\--timeout5m\--session-affinity=CLIENT_IP

5. Create atarget TCP proxy for the loadbalancer.

   If you want to turn on theproxy header,set it toPROXY_V1 instead ofNONE.

   gcloudbetacomputetarget-tcp-proxiescreatemy-tcp-lb-target-proxy\--backend-servicemy-tcp-lb\--proxy-headerNONE

6. Create aforwarding rule toroute traffic.

   This forwarding rule creates the load balancer.

   gcloudcomputeforwarding-rulescreatemy-tcp-lb-ipv4-forwarding-rule\--load-balancing-schemeEXTERNAL_MANAGED\--global\--target-tcp-proxymy-tcp-lb-target-proxy\--addresstcp-lb-static-ipv4\--ports80

Verify load balancer resources creation

You have created the following resources:

• An external virtual IP address
• The forwarding rule
• The firewall rule
• The target proxy
• The backend service
• The Compute Engine health check

The external virtual IP address is connected to the forwarding rule, whichdirects traffic that is allowed by the firewall rule to the target proxy. Thetarget proxy then communicates with the backend service, which is periodicallychecked by the health check. The relationship between these resources is shownin the following diagram:

These resources together are a load balancer. In the next step, you will addbackends to the load balancer.

One of the benefits of the standalone NEGs demonstrated here is that the lifecyclesof the load balancer and backend can be completely independent. The loadbalancer can continue running after the application, its services, or theGKE cluster is deleted. You can add and remove new NEGs ormultiple NEGs from the load balancer without changing any of the frontend loadbalancer objects.

Add standalone NEGs as backends to the load balancer

Usegcloud compute backend-servicesadd-backend toconnect the NEG to the load balancer by adding as a backend of themy-tcp-lb backend service:

gcloudcomputebackend-servicesadd-backendmy-tcp-lb\--global\--network-endpoint-group=NEG_NAME\--network-endpoint-group-zone=NEG_ZONE\--balancing-modeCONNECTION\--max-connections100

Replace the following:

• NEG_NAME: the name of your network endpointgroup. The name is either thename you specified when creating theNEG or an autogenerated name. If you did not specify a namefor the NEG, see the following instructions to find the autogenerated name.
• NEG_ZONE: the zone your networkendpoint group is in. See the following instructions to find this value.

To get the name and location of the NEGs, use this command:

gcloudcomputenetwork-endpoint-groupslist

The output is similar to the following:

NAME: k8s1-65a95e90-default-neg-demo-svc-80-663a85e4LOCATION: us-central1-aENDPOINT_TYPE: GCE_VM_IP_PORTSIZE: 3

In this example output, the name of the NEG iskk8s1-65a95e90-default-neg-demo-svc-80-663a85e4 and zone isus-central1-a.

Multiple NEGs can be added to the same backend service. Global backend serviceslikemy-tcp-lb can have NEG backends in different regions, while regionalbackend services must have backends in a single region.

Verify the load balancer configuration and connectivity

gcloudcomputebackend-servicesdescribemy-tcp-lb--global

backends:- balancingMode: CONNECTION  group: ... /networkEndpointGroups/k8s1-65a95e90-default-neg-demo-svc-80-663a85e4  maxConnections: 100...healthChecks:- ... /healthChecks/my-tcp-health-check...name: my-tcp-lb...

gcloudcomputebackend-servicesget-healthmy-tcp-lb--global

status:  healthStatus:  - healthState: HEALTHY    instance: ... gke-cluster-3-default-pool-4cc71a15-qlpf    ipAddress: 10.12.1.43    port: 50000  - healthState: HEALTHY    instance: ... gke-cluster-3-default-pool-4cc71a15-qlpf    ipAddress: 10.12.1.44    port: 50000  - healthState: HEALTHY    instance: ... gke-cluster-3-default-pool-4cc71a15-w9nk    ipAddress: 10.12.2.26    port: 50000

This examples extends theprevious example that uses an external HTTPload balancer.

Because all endpoints are grouped by the samebackendService, the VM andcontainer endpoints are considered thesame service. This means the host or pathmatching will treat all backends identically based on the URL map rules.

When you use a NEG as a backend for a backend service, all other backends inthat backend service must also be NEGs. You can't use instance groups and NEGsas backends in the same backend service. Additionally, containers and VMs cannotexist as endpoints within the same NEG, so they must always be configured withseparate NEGs.

1. Deploy a VM to Compute Engine with this command:

   gcloudcomputeinstancescreatevm1\--zone=COMPUTE_ZONE\--network=NETWORK\--subnet=SUBNET\--image-project=cos-cloud\--image-family=cos-stable--tags=vm-neg-tag

   Replace the following:

   • COMPUTE_ZONE: the name of the zone.
   • NETWORK: the name of the network.
   • SUBNET: the name of the subnet associated with thenetwork.

2. Deploy an application to the VM:

   gcloudcomputesshvm1\--zone=COMPUTE_ZONE\--command="docker run -d --rm --network=host registry.k8s.io/serve_hostname:v1.4 && sudo iptables -P INPUT ACCEPT"

   This command deploys to the VM the same example application used in theearlier example. For simplicity, the application is run as a Docker containerbut this is not essential. Theiptables command is required to allowfirewall access to the running container.

3. Validate that the application is serving on port 9376 and reportingthat it is running on vm1:

   gcloudcomputesshvm1\--zone=COMPUTE_ZONE\--command="curl -s localhost:9376"

   The server should respond withvm1.

4. Create a NEG to use with the VM endpoint. Containers and VMs can both be NEGendpoints, but a single NEG can't have both VM and container endpoints.

   gcloudcomputenetwork-endpoint-groupscreatevm-neg\--subnet=SUBNET\--zone=COMPUTE_ZONE

5. Attach the VM endpoint to the NEG:

   gcloudcomputenetwork-endpoint-groupsupdatevm-neg\--zone=COMPUTE_ZONE\--add-endpoint="instance=vm1,ip=VM_PRIMARY_IP,port=9376"

   ReplaceVM_PRIMARY_IP with the VM's primary IP address.

6. Confirm that the NEG has the VM endpoint:

   gcloudcomputenetwork-endpoint-groupslist-network-endpointsvm-neg\--zoneCOMPUTE_ZONE

7. Attach the NEG to the backend service using thesame command that you usedto add a container backend:

   gcloudcomputebackend-servicesadd-backendmy-bes--global\--network-endpoint-groupvm-neg\--network-endpoint-group-zoneCOMPUTE_ZONE\--balancing-modeRATE--max-rate-per-endpoint10

8. Open firewall to allow the VM's health check:

   gcloudcomputefirewall-rulescreatefw-allow-health-check-to-vm1\--network=NETWORK\--action=allow\--direction=ingress\--target-tags=vm-neg-tag\--source-ranges=130.211.0.0/22,35.191.0.0/16\--rules=tcp:9376

9. Validate that the load balancer is forwarding traffic to both the new vm1backend and the existing container backend by sending test traffic:

   foriin`seq1100`;docurl${VIP};echo;done

   You should see responses from both the container (neg-demo-app) and VM (vm1) endpoints.

VMs and containers for different backend services

This example shows how to create a NEG that points at an existing VM running aworkload, and how to add this NEG as the backend to a newbackendService. Thisis useful for the case where the containers and VMs are different services butneed to share the same L7 load balancer, such as if the services share the sameIP address or domain name.

This examples extends theprevious example that has a VM backend inthe same backend service as the container backend. This example reuses that VM.

Because the container and VM endpoints are grouped in separate backend services,they are considereddifferent services. This means that the URL map willmatch backends and direct traffic to the VM or container based on the hostname.

The following diagram shows how a single virtual IP address corresponds to twohost names, which in turn correspond to a container-based backend service and aVM-based backend service.

The following diagram shows the architecture described in the previous section:

1. Create a new backend service for the VM:

   gcloudcomputebackend-servicescreatemy-vm-bes\--protocolHTTP\--health-checkshttp-basic-check\--global

2. Attach the NEG for the VM,vm-neg, to the backend-service:

   gcloudcomputebackend-servicesadd-backendmy-vm-bes\--global\--network-endpoint-groupvm-neg\--network-endpoint-group-zoneCOMPUTE_ZONE\--balancing-modeRATE--max-rate-per-endpoint10

3. Add a host rule to theURL map todirect requests forcontainer.example.com host to the container backend service:

   gcloudcomputeurl-mapsadd-path-matcherweb-map\--path-matcher-name=container-path--default-service=my-bes\--new-hosts=container.example.com--global

4. Add another host rule the URL map to direct requests forvm.example.comhost to the VM backend service:

   gcloudcomputeurl-mapsadd-path-matcherweb-map\--path-matcher-name=vm-path--default-service=my-vm-bes\--new-hosts=vm.example.com--global

   Note: It takes a few minutes for the load balancer to be fully programmed.

5. Validate that the load balancer sends traffic to the VM backend based on therequested path:

   curl-H"HOST:vm.example.com"VIRTUAL_IP

   ReplaceVIRTUAL_IP with the virtual IP address.

Limitations of standalone NEGs

• Annotation validation errors are exposed to the user through Kubernetesevents.
• Thelimitations ofNEGsalso apply to standalone NEGs.
• Standalone NEGs don't work withlegacy networks.
• Standalone NEGs can only be used with compatible network services includingCloud Service Mesh and thecompatible load balancer types.

Pricing

Refer to theload balancing section of the pricing pagefor details on pricing of the load balancer. There is no additional charge forNEGs.

Troubleshooting

This section provides troubleshooting steps for common issues that you mightencounter with the standalone NEGs.

No standalone NEG configured

Symptom: No NEG is created.

Potential Resolution:

• Check the events associated with the Service and look for error messages.
• Verify the standalone NEG annotation is well-formed JSON, and that the exposedports match existing Ports in the Service spec.
• Verify the NEG status annotation and see if expected service ports hascorresponding NEGs.
• Verify that the NEGs have been created in the expected zones,with the commandgcloud compute network-endpoint-groups list.
• If using GKE version 1.18 or later, check if thesvcneg resource for the Service exists. If it does, check theInitialized condition for any error information.
• If you are usingcustom NEG names, ensure that each NEGname in is unique in its region.

Traffic does not reach the endpoints

Symptom: 502 errors or rejected connections.

Potential Resolution:

• After the service is configured new endpoints will generally become reachableafter attaching them to NEG, provided they respond to health checks.
• If after this time traffic still can't reach the endpoints resulting in 502error code for HTTP(S) or connections being rejected for TCP/SSL loadbalancers, check the following:
  • Verify that firewall rules allow incoming TCP traffic to your endpoints fromfollowing ranges:130.211.0.0/22 and35.191.0.0/16.
  • Verify that your endpoints are healthy using the Google Cloud CLI or by callinggetHealth API on thebackendService or the listEndpoints API on the NEGwith the showHealth parameter set toSHOW.

Stalled rollout

Symptom: Rolling out an updated Deployment stalls, and the number ofup-to-date replicas does not match the chosen number of replicas.

Potential Resolution:

The deployment's health checks are failing. The container image might be bad orthe health check might be misconfigured. The rolling replacement of Pods waitsuntil the newly started Pod passes its Pod Readiness gate. This only occurs ifthe Pod is responding to load balancer health checks. If the Pod does notrespond, or if the health check is misconfigured, the readiness gate conditionscan't be met and the rollout can't continue.

• If you're using kubectl 1.13 or higher, you can check the status of a Pod's readiness gates with the following command:

  kubectlgetmy-Pod-owide

  Check theREADINESS GATES column.

  This column doesn't exist in kubectl 1.12 and lower. A Pod that is marked asbeing in the READY state may have a failed readiness gate. To verify this, usethe following command:

  kubectlgetmy-pod-oyaml

  The readiness gates and their status are listed in the output.

• Verify that the container image in your Deployment's Pod specification isfunctioning correctly and is able to respond to health checks.

• Verify that the health checks are correctly configured.

NEG is not garbage collected

Symptom: A NEG that should have been deleted still exists.

Potential Resolution:

• The NEG is not garbage collected if the NEG is referenced by a backendservice. SeePreventing leaked NEGs for details.
• If using 1.18 or later, you can check for events in theServiceNetworkEndpointGroup resource usingthe service neg procedure.
• Check to see if the NEG is still needed by a service. Check thesvcnegresource for the service that corresponds to the NEG and check if a Serviceannotation exists.

NEG is not synced with Service

Symptom: Expected (Pod IP) endpoints don't exist in the NEG, the NEGis not synchronized, or the errorFailed to sync NEG_NAME (will not retry):neg name NEG_NAME is already in use, found a custom named neg with an emptydescription

Potential Resolution:

If you are using GKE 1.18 or later,check thesvcneg resourcefor information:

• Check thestatus.lastSyncTime value to verify if the NEG has syncedrecently.
• Check theSynced condition for any errors that occurred in the most recentsync.

If you are using GKE 1.19.9 or later, check whether thereexists a NEG whose name and zone match the name and zone of the NEG that theGKE NEG controller needs to create. For example, a NEGwith the name that the NEG controller needs to use might have been created usinggcloud CLI or the Google Cloud console in the cluster's zone (orone of the cluster's zones). In this case, you must delete the existing NEGbefore the NEG controller can synchronize its endpoints. Standalone NEG creationand membership are designed to be managed by the NEG controller.

What's next

• Zonal network endpoint groups overview
• Backend Services
• Creating Health Checks
• Using Target Proxies
• Using Forwarding Rules