This page shows you how to set up and use Ingress for internal Application Load Balancers inGoogle Kubernetes Engine (GKE). Ingress provides built-insupport for internal load balancing through the GKE Ingress controller.

To learn more about which features are supported for Ingress forinternal Application Load Balancers, seeIngress features.You can also learn more about how Ingress forinternal Application Load Balancers works inIngress for internal Application Load Balancers.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Requirements

Ingress for internal Application Load Balancers has the following requirements:

• Your cluster must use a GKE version later than 1.16.5-gke.10.
• Your cluster must beVPC-native.
• Your cluster must have theHttpLoadBalancing add-on enabled. This add-onis enabled by default. You must not disable it.
• You must useNetwork Endpoint Groups (NEGs) asbackends for your Service.

Deploying Ingress for internal Application Load Balancers

The following exercises show you how to deploy Ingress for internal Application Load Balancers:

1. Prepare your environment.
2. Create a cluster.
3. Deploy an application.
4. Deploy a Service.
5. Deploy Ingress.
6. Validate the deployment.
7. Delete Ingress resources.

Prepare your environment

Before you can deploy load balancer resources through the Kubernetes IngressAPI, you must prepare yournetworking environmentso that the load balancer proxies can be deployed in a given region.

Create a proxy-only subnet:

gcloudcomputenetworkssubnetscreateproxy-only-subnet\--purpose=REGIONAL_MANAGED_PROXY\--role=ACTIVE\--region=COMPUTE_REGION\--network=NETWORK_NAME\--range=10.129.0.0/23

Replace the following:

• COMPUTE_REGION: aCompute Engine region.
• NETWORK_NAME: the name of the network for thesubnet.

For more information, seeconfiguring the proxy-only subnet.

Create a firewall rule

The Ingress controller does not create a firewall rule to allowconnections from the load balancer proxies in the proxy-subnet. You must createthis firewall rule manually. However, the Ingress controller does createfirewall rules to allow ingress for Google Cloud health-checks.

Create a firewall rule to allow connections from the load balancer proxies inthe proxy-only subnet to the pod listening port:

gcloudcomputefirewall-rulescreateallow-proxy-connection\--allow=TCP:CONTAINER_PORT\--source-ranges=10.129.0.0/23\--network=NETWORK_NAME

ReplaceCONTAINER_PORT with the value of the port thatthe Pod is listening to, such as9376.

Creating a cluster

In this section, you create a VPC-native cluster that you canuse with Ingress for internal Application Load Balancers. You can create this cluster usingthe Google Cloud CLI or the Google Cloud console.

gcloudcontainerclusterscreate-autoCLUSTER_NAME\--location=COMPUTE_LOCATION\--network=NETWORK_NAME

Deploying a web application

In this section, you create aDeployment.

To create a Deployment:

1. Save the following sample manifest asweb-deployment.yaml:

   apiVersion:apps/v1kind:Deploymentmetadata:labels:app:hostnamename:hostname-serverspec:selector:matchLabels:app:hostnameminReadySeconds:60replicas:3template:metadata:labels:app:hostnamespec:containers:-image:registry.k8s.io/serve_hostname:v1.4name:hostname-serverports:-containerPort:9376protocol:TCPterminationGracePeriodSeconds:90

   This manifest describes a Deployment that listens on an HTTPS server onport 9376. This Deployment also manages Pods for your application. Each Podruns one application container with an HTTPS server that returns the hostnameof the application server as the response. The default hostname of a Pod isthe name of the Pod. The container also handles graceful termination.

2. Apply the manifest to the cluster:

   kubectlapply-fweb-deployment.yaml

Deploying a Service as a Network Endpoint Group (NEG)

In this section, you create aServiceresource. The Service selects the backend containers by their labels so that theIngress controller can program them as backend endpoints. Ingress forinternal Application Load Balancers requires you to useNEGs asbackends. The feature does not support Instance Groups as backends. Because NEGbackends are required, the following NEG annotation is required when you deployServices that are exposed through Ingress:

annotations:cloud.google.com/neg:'{"ingress":true}'

Your Service is automatically annotated withcloud.google.com/neg: '{"ingress": true}' when all of the followingconditions are true:

• You are using VPC-native clusters.
• You are not using a Shared VPC.
• You are not usingGKE Network Policy.

The annotation is automatically added using aMutatingWebhookConfiguration with nameneg-annotation.config.common-webhooks.networking.gke.io. You can check if theMutatingWebhookConfiguration is present with the following command:

kubectlgetmutatingwebhookconfigurations

The usage of NEGs allows the Ingress controller to performcontainer native load balancing.Traffic is load balanced from the Ingress proxy directly to the PodIP as opposed to traversing the node IP or kube-proxy networking. In addition,Pod readiness gatesare implemented to determine the health of Pods from theperspective of the load balancer and not only the Kubernetes readiness andliveness checks. Pod readiness gates ensure that traffic is not dropped duringlifecycle events such as Pod startup, Pod loss, or node loss.

If you do not include a NEG annotation, you receive a warning on the Ingress objectthat prevents you from configuring the internal Application Load Balancer. A Kubernetes event isalso generated on the Ingress if the NEG annotation is not included. Thefollowing message is an example of the event message:

Message-------error while evaluating the ingress spec: could not find port "8080" in service "default/no-neg-svc"

An NEG is not created until an Ingress references the Service. The NEG does notappear in Compute Engine until the Ingress and its referenced Service bothexist. NEGs are a zonal resource and for multi-zonal clusters, one iscreated per Service per zone.

To create a Service:

1. Save the following sample manifest asweb-service.yaml:

   apiVersion:v1kind:Servicemetadata:name:hostnamenamespace:defaultannotations:cloud.google.com/neg:'{"ingress":true}'spec:ports:-name:host1port:80protocol:TCPtargetPort:9376selector:app:hostnametype:ClusterIP

2. Apply the manifest to the cluster:

   kubectlapply-fweb-service.yaml

Deploying Ingress

In this section, you create an Ingress resource that triggers the deployment ofa Compute Engine load balancer through the Ingress controller. Ingress forinternal Application Load Balancers requires the following annotation:

annotations:kubernetes.io/ingress.class:"gce-internal"

You cannot use theingressClassName field to specify a GKEIngress. You must use thekubernetes.io/ingress.class annotation. For moreinformation, seeGKE Ingress controller behavior.

To create an Ingress:

1. Save the following sample manifest asinternal-ingress.yaml:

   apiVersion:networking.k8s.io/v1kind:Ingressmetadata:name:ilb-demo-ingressnamespace:defaultannotations:kubernetes.io/ingress.class:"gce-internal"spec:defaultBackend:service:name:hostnameport:number:80

2. Apply the manifest to the cluster:

   kubectlapply-finternal-ingress.yaml

Validating a successful Ingress deployment

In this section, you validate if your deployment was successful.

It can take several minutes for the Ingress resource to become fullyprovisioned. During this time, the Ingress controller creates items such asforwarding rules, backend services, URL maps, and NEGs.

To retrieve the status of your Ingress resource that you created in the previoussection, run the following command:

kubectlgetingressilb-demo-ingress

The output is similar to the following:

NAME               HOSTS    ADDRESS            PORTS     AGEilb-demo-ingress   *        10.128.0.58        80        59s

When theADDRESS field is populated, the Ingress is ready. The use of an RFC1918 address in this field indicates an internal IP within theVPC.

Since the internal Application Load Balancer is a regional load balancer, thevirtual IP (VIP) is only accessible from a client within the same region andVPC. After retrieving the load balancer VIP, you can use tools(for example,curl) to issueHTTP GET calls against the VIP from inside theVPC.

To issue aHTTP GET call, complete the following steps:

1. To reach your Ingress VIP from inside the VPC, deploy a VMwithin the same region and network as the cluster:

   gcloudcomputeinstancescreatel7-ilb-client\--image-family=debian-12\--image-project=debian-cloud\--network=NETWORK_NAME\--subnet=SUBNET_NAME\--zone=COMPUTE_ZONE\--tags=allow-ssh

   Replace the following:

   • SUBNET_NAME: the name of a subnet in the network.
   • COMPUTE_ZONE: aCompute Engine zonein the region.

   To learn more about creating instances, seeCreating and starting a VM instance.

2. To access the internal VIP from inside the VM, usecurl:

   1. SSH in to the VM that you created in the previous step:

      gcloudcomputesshl7-ilb-client\--zone=COMPUTE_ZONE

   2. Usecurl to access the internal application VIP:

      curl10.128.0.58hostname-server-6696cf5fc8-z4788

      The successful HTTP response and hostname of one of the backend containersindicates that the full load balancing path is functioning correctly.

Deleting Ingress resources

Removing Ingress and Service resources also removes the Compute Engine loadbalancer resources associated with them. To prevent resource leaking,ensure that Ingress resources are torn down when you no longer need them. Youmust also delete Ingress and Service resources before you delete clusters orelse the Compute Engine load balancing resources are orphaned.

To remove an Ingress, complete the following steps:

1. Delete the Ingress. For example, to delete the Ingress you created in thispage, run the following command:

   kubectldeleteingressilb-demo-ingress

   Deleting the Ingress removes the forwarding rules, backend services, and URLmaps associated with this Ingress resource.

2. Delete the Service. For example, to delete the Service you created in thispage, run the following command:

   kubectldeleteservicehostname

   Deleting the Service removes the NEG associated with the Service.

To deploy an application on GKE and expose the application with a private loadbalanced IP address, seeBasic Internal Ingress.

Static IP addressing

Internal Ingress resources support both static and ephemeral IP addressing. Ifan IP address is not specified, an available IP address is automaticallyallocated from the GKE node subnet. However, the Ingress resourcedoes not provision IP addresses from theproxy-only subnetas that subnet is only used for internal proxy consumption. These ephemeral IPaddresses are allocated to the Ingress only for the lifecycle of the internal Ingressresource. If you delete your Ingress and create a new Ingress from the samemanifest file, you are not guaranteed to get the same external IP address.

If you want a permanent IP address that's independent from the lifecycle of theinternal Ingress resource, you must reserve a regional static internal IPaddress. You can then specify a static IP address by using thekubernetes.io/ingress.regional-static-ip-name annotation on your Ingressresource.

The following example shows you how to add this annotation:

apiVersion:networking.k8s.io/v1kind:Ingressmetadata:name:my-ingressannotations:kubernetes.io/ingress.regional-static-ip-name:STATIC_IP_NAMEkubernetes.io/ingress.class:"gce-internal"

ReplaceSTATIC_IP_NAME with a static IP name that meetsthe following criteria:

• Create the static IP address before you deploy the Ingress. A load balancerdoes not deploy until the static IP exists, and referencing a non-existentIP address resource does not create a static IP. If you modify an existingIngress to use a static IP address instead of an ephemeral IP address,GKE might change the IP address of the load balancer whenGKE re-creates the forwarding rule of the load balancer.
• The static IP isreserved in the service projectfor an Ingress deployed in the service project of a Shared VPC.
• Reference the Google Cloud IP address resource by its name, rather than itsIP address.
• The IP address must be from a subnet in the same region as theGKE cluster. You can use any available private subnet withinthe region (with the exception of theproxy-only subnet).Different Ingress resources can also have addresses from different subnets.

HTTPS between client and load balancer

Ingress for internal load balancing supports the serving of TLS certificates toclients. You can serve TLS certificates through Kubernetes Secrets or throughpre-shared regional SSL certificates in Google Cloud. You can also specifymultiple certificates per Ingress resource.Use of both HTTPS and HTTP simultaneously is supported for GKE 1.25+. To enablethis feature, you need to create a static IP address with PURPOSE=SHARED_LOADBALANCER_VIP,and configure it on the ingress. If a static IP address is not provided, only HTTPStraffic is allowed, and you need to follow the documentation forDisabling HTTP.

1. Create the regional certificate:

   gcloudcomputessl-certificatescreateCERT_NAME\--certificateCERT_FILE_PATH\--private-keyKEY_FILE_PATH\--regionCOMPUTE_REGION

   Replace the following:

   • CERT_NAME: a name for your certificate that youchoose.
   • CERT_FILE_PATH: the path to your local certificatefile to create a self-managed certificate. The certificate must be in PEMformat.
   • KEY_FILE_PATH: the path to a local private key file.The private key must be in PEM format and must use RSA or ECDSA encryption.
   • COMPUTE_REGION: a Compute Engine region for yourcertificate.

2. Reserve and apply a static IP address followingStatic IP addressing.

3. Save the following sample manifest asingress-pre-shared-cert.yaml:

   apiVersion:networking.k8s.io/v1kind:Ingressmetadata:name:ilb-demo-ingnamespace:defaultannotations:ingress.gcp.kubernetes.io/pre-shared-cert:"CERT_NAME"kubernetes.io/ingress.regional-static-ip-name:STATIC_IP_NAMEkubernetes.io/ingress.class:"gce-internal"spec:rules:-host:DOMAINhttp:paths:-pathType:ImplementationSpecificbackend:service:name:SERVICE_NAMEport:number:80

   Replace the following:

   • DOMAIN: your domain.
   • CERT_NAME: the name of the certificate you createdin the previous section.
   • SERVICE_NAME: the name of your Service.

4. Apply the manifest to the cluster:

   kubectlapply-fingress-pre-shared-cert.yaml

HTTPS between load balancer and application

If your application runs in a GKE Pod and can receiveHTTPS requests, you can configure the load balancer to use HTTPS when it forwardsrequests to your application. For more information, seeHTTPS (TLS) between load balancer and your application.

Shared VPC

Manually add the NEG annotation

If the GKE in which you are deploying the Ingress resources is in aShared VPC service project, the services are not automatically annotated withthe annotationcloud.google.com/neg: '{"ingress": true}' because theMutatingWebhookConfiguration responsible for injecting the annotation to theservices is not installed.

You must add the NEG annotation to the manifest of the Services that areexposed through Ingress for internal Application Load Balancers.

VPC firewall rules

If the GKE cluster in which you are deploying the Ingress resources isin a Shared VPC service project, and you want the GKE controlplane to manage the firewall resources in your host project, then the service project'sGKE service account must be granted the appropriate IAMpermissions in the host project as perManaging firewall resources for clusters with Shared VPC.This lets the Ingress controller create firewall rules to allow ingress traffic forGoogle Cloud health checks.

The following is an example of an event that might be present in the Ingress resource logs. This error occurs when the Ingress controller is unable to createa firewall rule to allow ingress traffic for Google Cloud health checks if the permissions are not configured correctly.

Firewall change required by security admin: `gcloud compute firewall-rules update <RULE_NAME> --description "GCE L7 firewall rule" --allow tcp:<PORT> --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags <TARGET_TAG> --project <HOST_PROJECT>

If you prefer tomanually provision firewall rules from the host project, then you can mute thefirewallXPNError eventsby adding thenetworking.gke.io/suppress-firewall-xpn-error: "true" annotation to the Ingress resource.

Summary of internal Ingress annotations

The following tables show you the annotations that you can add when you arecreating Ingress and Service resources for Ingress for internal Application Load Balancers.

Ingress annotations

Service annotations related to Ingress

Troubleshooting

Understanding and observing the state of Ingress typically involves inspectingthe associated resources. The types of issues encountered often include loadbalancing resources not being created properly, traffic not reaching backends,or backends not appearing healthy.

Some common troubleshooting steps include:

• Verifying that client traffic is originating from within the same region andVPC as the load balancer.
• Verifying that the Pods and backends are healthy.
• Validating the traffic path to the VIP and for Compute Engine health checksto ensure it is not blocked by firewall rules.
• Checking the Ingress resource events for errors.
• Describing the Ingress resource to see the mapping to Compute Engineresources.
• Validating that the Compute Engine load balancing resources exist, have thecorrect configurations, and do not have errors reported.

Filtering for Ingress events

The following query filters for errors across all Ingress events in yourcluster:

kubectlgetevents--all-namespaces--field-selectorinvolvedObject.kind=Ingress

You can also filter by objects or object names:

kubectlgetevents--field-selectorinvolvedObject.kind=Ingress,involvedObject.name=hostname-internal-ingress

In the following error, you can see that the Service referenced by the Ingressdoes not exist:

LAST SEEN   TYPE      REASON      OBJECT                              MESSAGE0s          Warning   Translate   ingress/hostname-internal-ingress   error while evaluating the ingress spec: could not find service "default/hostname-invalid"

Inspecting Compute Engine load balancer resources

The following command displays the full output for the Ingress resource so thatyou can see the mappings to the Compute Engine resources that are created bythe Ingress controller:

kubectlgetingINGRESS_FILENAME-oyaml

ReplaceINGRESS_FILENAME with your Ingress resource'sfilename.

The output is similar to the following:

apiVersion: v1items:- apiVersion: networking.k8s.io/v1  kind: Ingress  metadata:    annotations:ingress.kubernetes.io/backends: '{"k8s1-241a2b5c-default-hostname-80-29269aa5":"HEALTHY"}'      ingress.kubernetes.io/forwarding-rule: k8s-fw-default-ilb-demo-ingress--241a2b5c94b353ec      ingress.kubernetes.io/target-proxy: k8s-tp-default-ilb-demo-ingress--241a2b5c94b353ec      ingress.kubernetes.io/url-map: k8s-um-default-ilb-demo-ingress--241a2b5c94b353ec      kubectl.kubernetes.io/last-applied-configuration: |       {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"gce-internal"},"name":"ilb-demo-ingress","namespace":"default"},"spec":{"defaultBackend":{"service":{"name":"hostname"},"port":{"number":80}}}}      kubernetes.io/ingress.class: gce-internal    creationTimestamp: "2019-10-15T02:16:18Z"    finalizers:    - networking.gke.io/ingress-finalizer    generation: 1    name: ilb-demo-ingress    namespace: default    resourceVersion: "1538072"    selfLink: /apis/networking.k8s.io/v1/namespaces/default/ingresses/ilb-demo-ingress    uid: 0ef024fe-6aea-4ee0-85f6-c2578f554975  spec:    defaultBackend:      service:        name: hostname        port:          number: 80  status:    loadBalancer:      ingress:      - ip: 10.128.0.127kind: Listmetadata:  resourceVersion: ""  selfLink: ""

Theingress.kubernetes.io/backends annotations list the backends and theirstatus. Make sure that your backends are listed asHEALTHY.

The Compute Engine resources created by the Ingress can be queried directlyto understand their status and configuration. Running these queries can also behelpful when troubleshooting.

To list all Compute Engine forwarding rules:

gcloudcomputeforwarding-ruleslist

The output is similar to the following:

NAME                                                        REGION       IP_ADDRESS      IP_PROTOCOL  TARGETk8s-fw-default-hostname-internal-ingress--42084f6a534c335b  REGION_NAME  10.128.15.225   TCP          REGION_NAME/targetHttpProxies/k8s-tp-default-hostname-internal-ingress--42084f6a534c335b

To list the health of a backend service, first list the backend services, andmake a copy of the name of the backend service you want to inspect:

gcloudcomputebackend-serviceslist

The output is similar to the following:

NAME                                         BACKENDS                                                                       PROTOCOLk8s1-42084f6a-default-hostname-80-98cbc1c1   REGION_NAME/networkEndpointGroups/k8s1-42084f6a-default-hostname-80-98cbc1c1 HTTP

You can now use the backend service name to query its health:

gcloudcomputebackend-servicesget-healthk8s1-42084f6a-default-hostname-80-98cbc1c1\--regionCOMPUTE_REGION

ReplaceCOMPUTE_REGION with the Compute Engineregion of the backend service.

The output is similar to the following:

backend: https://www.googleapis.com/compute/v1/projects/user1-243723/zones/ZONE_NAME/networkEndpointGroups/k8s1-42084f6a-default-hostname-80-98cbc1c1status:  healthStatus:  - healthState: HEALTHY

What's next

• Learn aboutGKE Ingress for external Application Load Balancers.

• Read a conceptual overview ofServices in GKE.

• Learn how to create aninternal passthrough Network Load Balancer on GKE.

• Implement abasic internal Ingress.