Use UDP with external passthrough Network Load Balancers
This document discusses how to work withexternal passthrough Network Load Balancers by using the User Datagram Protocol (UDP). The document is intended for appdevelopers, app operators, and network administrators.
About UDP
UDP is used commonly in apps. The protocol, which is described inRFC-768, implements a stateless, unreliable datagram packet service.For example, Google'sQUIC protocol improves the user experience by using UDP to speed up stream-basedapps.
The stateless part of UDP means that the transport layer doesn't maintain astate. Therefore, each packet in a UDP "connection" is independent. In fact,there is no real connection in UDP. Instead, its participants usually use a2-tuple (ip:port) or a 4-tuple (src-ip:src-port,dest-ip:dest-port) torecognize each other.
Like TCP-based apps, UDP-based apps can also benefit from a load balancer,which is why external passthrough Network Load Balancers are used in UDP scenarios.
External passthrough Network Load Balancer
External passthrough Network Load Balancers are passthrough load balancers; theyprocess incoming packets and deliver them to backend servers with the packetsintact. The backend servers then send the returning packets directly to theclients. This technique is called Direct Server Return (DSR). On each Linuxvirtual machine (VM) running on Compute Engine that is a backend of aGoogle Cloud external passthrough Network Load Balancer, an entry in the local routing table routestraffic that's destined for the load balancer's IP address to the networkinterface controller (NIC). The following example demonstrates this technique:
root@backend-server:~#iprolstablelocallocal10.128.0.2deveth0protokernelscopehostsrc10.128.0.2broadcast10.128.0.2deveth0protokernelscopelinksrc10.128.0.2local198.51.100.2deveth0proto66scopehostbroadcast127.0.0.0devloprotokernelscopelinksrc127.0.0.1local127.0.0.0/8devloprotokernelscopehostsrc127.0.0.1local127.0.0.1devloprotokernelscopehostsrc127.0.0.1broadcast127.255.255.255devloprotokernelscopelinksrc127.0.0.1In the preceding example,198.51.100.2 is the load balancer's IP address. Thegoogle-network-daemon.service agent is responsible for adding this entry.However, as the following example shows, the VM does not actually have aninterface that owns the load balancer's IP address:
root@backend-server:~#ipadls1:lo:<LOOPBACK,UP,LOWER_UP>mtu65536qdiscnoqueuestateUNKNOWNgroupdefaultqlen1link/loopback00:00:00:00:00:00brd00:00:00:00:00:00inet127.0.0.1/8scopehostlovalid_lftforeverpreferred_lftforeverinet6::1/128scopehostvalid_lftforeverpreferred_lftforever2:eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1460qdiscmqstateUPgroupdefaultqlen1000link/ether42:01:0a:80:00:02brdff:ff:ff:ff:ff:ffinet10.128.0.2/32brd10.128.0.2scopeglobaleth0valid_lftforeverpreferred_lftforeverinet6fe80::4001:aff:fe80:2/64scopelinkvalid_lftforeverpreferred_lftforeverThe external passthrough Network Load Balancer transmits the incoming packets, with the destinationaddress untouched, to the backend server. The local routing table entry routesthe packet to the correct app process, and the response packets from the app aresent directly to the client.
The following diagram shows how external passthrough Network Load Balancers work. Theincoming packets are processed by a load balancer calledMaglev,which distributes the packets to the backend servers. Outgoing packets are thensent directly to the clients through DSR.
An issue with UDP return packets
When you work with DSR, there is a slight difference between how the Linuxkernel treats TCP and UDP connections. Because TCP is a stateful protocol, thekernel has all the information it needs about the TCP connection, including theclient address, client port, server address, and server port. This informationis recorded in the socket data structure that represents the connection. Thus,each returning packet of a TCP connection has the source address correctly setto the server address. For a load balancer, that address is the load balancer'sIP address.
Recall that UDP is stateless, however, so the socket objects that are createdin the app process for UDP connections don't have the connection information.The kernel doesn't have the information about the source address of an outgoingpacket, and it doesn't know the relation to a previously received packet. Forthe packet's source address, the kernel can only fill in the address of theinterface that the returning UDP packet goes to. Or if the app previously boundthe socket to a certain address, the kernel uses that address as the sourceaddress.
The following code shows a simple echo program:
#!/usr/bin/python3importsocket,structdefloop_on_socket(s):whileTrue:d,addr=s.recvfrom(1500)print(d,addr)s.sendto("ECHO: ".encode('utf8')+d,addr)if__name__=="__main__":HOST,PORT="0.0.0.0",60002sock=socket.socket(type=socket.SocketKind.SOCK_DGRAM)sock.bind((HOST,PORT))loop_on_socket(sock)Following is thetcpdump output during a UDP conversation:
14:50:04.758029 IP 203.0.113.2.40695 > 198.51.100.2.60002: UDP, length 314:50:04.758396 IP 10.128.0.2.60002 > 203.0.113.2.40695: UDP, length 2T
198.51.100.2 is the load balancer's IP address, and203.0.113.2 is theclient IP address.
After the packets leave the VM, another NAT device–a Compute Enginegateway–in the Google Cloud network translates the source address to theexternal address. The gateway doesn't know which external address should beused, so only the VM's external address (not the load balancer's) can be used.
From the client side, if you check the output fromtcpdump, the packets fromthe server look like the following:
23:05:37.072787 IP 203.0.113.2.40695 > 198.51.100.2.60002: UDP, length 523:05:37.344148 IP 198.51.100.3.60002 > 203.0.113.2.40695: UDP, length 4
198.51.100.3 is the VM's external IP address.
From the client's point of view, the UDP packets are not coming from an addressthat the client sent them to. This causes problems: the kernel drops thesepackets, and if the client is behind a NAT device, so does the NAT device. As aresult, the client app gets no response from the server. The following diagramshows this process where the client rejects returning packets because of addressmismatches.
Solving the UDP problem
To solve the no-response problem, you must rewrite the source address ofoutgoing packets to the load balancer's IP address at the server that's hostingthe app. Following are several options that you can use to accomplish thisheader rewrite. The first solution uses a Linux-based approach withiptables;the other solutions take app-based approaches.
The following diagram shows the core idea of these options: rewrite thesource IP address of the returning packets in order to match the load balancer'sIP address.
Use NAT policy in the backend server
The NAT policy solution is to use the Linuxiptables command to rewrite thedestination address from the load balancer's IP address to the VM's IP address.In the following example, you add aniptables DNAT rule to change thedestination address of the incoming packets:
iptables-tnat-APOSTROUTING-jRETURN-d10.128.0.2-pudp--dport60002iptables-tnat-APREROUTING-jDNAT--to-destination10.128.0.2-d198.51.100.2-pudp--dport60002This command adds two rules to the NAT table of theiptables system. Thefirst rule bypasses all incoming packets that target the localeth0 address.As a result, traffic that doesn't come from the load balancer isn't affected.The second rule changes the destination IP address of incoming packets to theVM's internal IP address. The DNAT rules are stateful, which means that thekernel tracks the connections and rewrites the returning packets' source addressautomatically.
| Pros | Cons |
|---|---|
| The kernel translates the address, with no change required to apps. | Extra CPU is used to do the NAT. And because DNAT is stateful, memory consumption might also be high. |
| Supports multiple load balancers. |
Usenftables to statelessly mangle the IP header fields
In thenftables solution, you use thenftables command to mangle the sourceaddress in the IP header of outgoing packets. This mangling is stateless, so itconsumes fewer resources than using DNAT. To usenftables, you need a Linuxkernel version greater than 4.10.
You use the following commands:
nftaddtablerawnftaddchainrawpostrouting{typefilterhookpostroutingpriority300)nftaddrulerawpostroutingipsaddr10.128.0.2udpsport60002ipsaddrset198.51.100.2| Pros | Cons |
|---|---|
| The kernel translates the address, with no change required to apps. | Does not support multiple load balancers. |
| The address translation process is stateless, so resource consumption is much lower. | Extra CPU is used to do the NAT. |
nftables are available only to newer Linux kernel versions. Some distros, like Centos 7.x, cannot usenftables. |
Let the app explicitly bind to the load balancer's IP address
In the binding solution, you modify your app so that it binds explicitly to theload balancer's IP address. For a UDP socket, thebind operation lets thekernel know which address to use as the source address when sending UDP packetsthat use that socket.
The following example shows how to bind to a specific address in Python:
#!/usr/bin/python3importsocketdefloop_on_socket(s):whileTrue:d,addr=s.recvfrom(1500)print(d,addr)s.sendto("ECHO: ".encode('utf8')+d,addr)if__name__=="__main__":# Instead of setting HOST to "0.0.0.0",# we set HOST to the Load Balancer IPHOST,PORT="198.51.100.2",60002sock=socket.socket(type=socket.SocketKind.SOCK_DGRAM)sock.bind((HOST,PORT))loop_on_socket(sock)# 198.51.100.2 is the load balancer's IP address# You can also use the DNS name of the load balancer's IP addressThe preceding code is a UDP server; it echoes back the bytes received, with apreceding"ECHO: ". Pay attention to lines 12 and 13, where the serveris bound to the address198.51.100.2, which is the load balancer's IP address.
| Pros | Cons |
|---|---|
| Can be achieved with a simple code change to the app. | Does not support multiple load balancers. |
Userecvmsg/sendmsg instead ofrecvfrom/sendto to specify the address
In this solution, you userecvmsg/sendmsg calls instead ofrecvfrom/sendto calls. In comparison torecvfrom/sendto calls, therecvmsg/sendmsg calls can handle ancillary control messages along with thepayload data. These ancillary control messages include the source or destinationaddress of the packets. This solution lets you fetch destination addresses fromincoming packets, and because those addresses are real load balanceraddresses, you can use them as source addresses when sending replies.
The following example program demonstrates this solution:
#!/usr/bin/python3importsocket,structdefloop_on_socket(s):whileTrue:d,ctl,flg,addr=s.recvmsg(1500,1024)# ctl contains the destination address informations.sendmsg(["ECHO: ".encode("utf8"),d],ctl,0,addr)if__name__=="__main__":HOST,PORT="0.0.0.0",60002s=socket.socket(type=socket.SocketKind.SOCK_DGRAM)s.setsockopt(0,# level is 0 (IPPROTO_IP)8,# optname is 8 (IP_PKTINFO)1)s.bind((HOST,PORT))loop_on_socket(s)This program demonstrates how to userecvmsg/sendmsg calls. In order tofetch address information from packets, you must use thesetsockopt call toset theIP_PKTINFO option.
| Pros | Cons |
|---|---|
| Works even if there are multiple load balancers–for example, when there are both internal and external load balancers configured to the same backend. | Requires you to make complex changes to the app. In some cases, this might not be possible. |
What's next
- Learn how to configure an external passthrough Network Load Balancer and distribute trafficinSet up an external passthrough Network Load Balancer.
- Read more aboutexternal passthrough Network Load Balancers.
- Read more about theMaglev technique behind external passthrough Network Load Balancers.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-12-15 UTC.