Overview¶

AF_XDP is an address family that is optimized for high performancepacket processing.

This document assumes that the reader is familiar with BPF and XDP. Ifnot, the Cilium project has an excellent reference guide athttp://cilium.readthedocs.io/en/latest/bpf/.

Using the XDP_REDIRECT action from an XDP program, the program canredirect ingress frames to other XDP enabled netdevs, using thebpf_redirect_map() function. AF_XDP sockets enable the possibility forXDP programs to redirect frames to a memory buffer in a user-spaceapplication.

An AF_XDP socket (XSK) is created with the normal socket()syscall. Associated with each XSK are two rings: the RX ring and theTX ring. A socket can receive packets on the RX ring and it can sendpackets on the TX ring. These rings are registered and sized with thesetsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatoryto have at least one of these rings for each socket. An RX or TXdescriptor ring points to a data buffer in a memory area called aUMEM. RX and TX can share the same UMEM so that a packet does not haveto be copied between RX and TX. Moreover, if a packet needs to be keptfor a while due to a possible retransmit, the descriptor that pointsto that packet can be changed to point to another and reused rightaway. This again avoids copying data.

The UMEM consists of a number of equally sized chunks. A descriptor inone of the rings references a frame by referencing its addr. The addris simply an offset within the entire UMEM region. The user spaceallocates memory for this UMEM using whatever means it feels is mostappropriate (malloc, mmap, huge pages, etc). This memory area is thenregistered with the kernel using the new setsockopt XDP_UMEM_REG. TheUMEM also has two rings: the FILL ring and the COMPLETION ring. TheFILL ring is used by the application to send down addr for the kernelto fill in with RX packet data. References to these frames will thenappear in the RX ring once each packet has been received. TheCOMPLETION ring, on the other hand, contains frame addr that thekernel has transmitted completely and can now be used again by userspace, for either TX or RX. Thus, the frame addrs appearing in theCOMPLETION ring are addrs that were previously transmitted using theTX ring. In summary, the RX and FILL rings are used for the RX pathand the TX and COMPLETION rings are used for the TX path.

The socket is then finally bound with abind() call to a device and aspecific queue id on that device, and it is not until bind iscompleted that traffic starts to flow.

The UMEM can be shared between processes, if desired. If a processwants to do this, it simply skips the registration of the UMEM and itscorresponding two rings, sets the XDP_SHARED_UMEM flag in the bindcall and submits the XSK of the process it would like to share UMEMwith as well as its own newly created XSK socket. The new process willthen receive frame addr references in its own RX ring that point tothis shared UMEM. Note that since the ring structures aresingle-consumer / single-producer (for performance reasons), the newprocess has to create its own socket with associated RX and TX rings,since it cannot share this with the other process. This is also thereason that there is only one set of FILL and COMPLETION rings perUMEM. It is the responsibility of a single process to handle the UMEM.

How is then packets distributed from an XDP program to the XSKs? Thereis a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). Theuser-space application can place an XSK at an arbitrary place in thismap. The XDP program can then redirect a packet to a specific index inthis map and at this point XDP validates that the XSK in that map wasindeed bound to that device and ring number. If not, the packet isdropped. If the map is empty at that index, the packet is alsodropped. This also means that it is currently mandatory to have an XDPprogram loaded (and one XSK in the XSKMAP) to be able to get anytraffic to user space through the XSK.

AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If thedriver does not have support for XDP, or XDP_SKB is explicitly chosenwhen loading the XDP program, XDP_SKB mode is employed that uses SKBstogether with the generic XDP support and copies out the data to userspace. A fallback mode that works for any network device. On the otherhand, if the driver has support for XDP, it will be used by the AF_XDPcode to provide better performance, but there is still a copy of thedata into user space.

Concepts¶

In order to use an AF_XDP socket, a number of associated objects needto be setup. These objects and their options are explained in thefollowing sections.

For an overview on how AF_XDP works, you can also take a look at theLinux Plumbers paper from 2018 on the subject:http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. DoNOT consult the paper from 2017 on “AF_PACKET v4”, the first attemptat AF_XDP. Nearly everything changed since then. Jonathan Corbet hasalso written an excellent article on LWN, “Accelerating networkingwith AF_XDP”. It can be found athttps://lwn.net/Articles/750845/.

Libbpf¶

Libbpf is a helper library for eBPF and XDP that makes using thesetechnologies a lot simpler. It also contains specific helper functionsin tools/testing/selftests/bpf/xsk.h for facilitating the use ofAF_XDP. It contains two types of functions: those that can be used tomake the setup of AF_XDP socket easier and ones that can be used in thedata plane to access the rings safely and quickly.

We recommend that you use this library unless you have become a poweruser. It will make your program a lot simpler.

XSKMAP / BPF_MAP_TYPE_XSKMAP¶

On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) thatis used in conjunction withbpf_redirect_map() to pass the ingressframe to a socket.

The user application inserts the socket into the map, via thebpf()system call.

Note that if an XDP program tries to redirect to a socket that doesnot match the queue configuration and netdev, the frame will bedropped. E.g. an AF_XDP socket is bound to netdev eth0 andqueue 17. Only the XDP program executing for eth0 and queue 17 willsuccessfully pass data to the socket. Please refer to the sampleapplication (samples/bpf/) in for an example.

Configuration Flags and Socket Options¶

These are the various configuration flags that can be used to controland monitor the behavior of AF_XDP sockets.

#include<linux/bpf.h>#include"bpf_helpers.h"#define MAX_SOCKS 16struct{__uint(type,BPF_MAP_TYPE_XSKMAP);__uint(max_entries,MAX_SOCKS);__uint(key_size,sizeof(int));__uint(value_size,sizeof(int));}xsks_mapSEC(".maps");staticunsignedintrr;SEC("xdp_sock")intxdp_sock_prog(structxdp_md*ctx){rr=(rr+1)&(MAX_SOCKS-1);returnbpf_redirect_map(&xsks_map,rr,XDP_DROP);}

if(xsk_ring_prod__needs_wakeup(&my_tx_ring))sendto(xsk_socket__fd(xsk_handle),NULL,0,MSG_DONTWAIT,NULL,0);

structxdp_statistics{__u64rx_dropped;/* Dropped for reasons other than invalid desc */__u64rx_invalid_descs;/* Dropped due to invalid descriptor */__u64tx_invalid_descs;/* Dropped due to invalid descriptor */};

Multi-Buffer Support¶

With multi-buffer support, programs using AF_XDP sockets can receiveand transmit packets consisting of multiple buffers both in copy andzero-copy mode. For example, a packet can consist of twoframes/buffers, one with the header and the other one with the data,or a 9K Ethernet jumbo frame can be constructed by chaining togetherthree 4K frames.

Some definitions:

• A packet consists of one or more frames

• A descriptor in one of the AF_XDP rings always refers to a singleframe. In the case the packet consists of a single frame, thedescriptor refers to the whole packet.

To enable multi-buffer support for an AF_XDP socket, use the new bindflag XDP_USE_SG. If this is not provided, all multi-buffer packetswill be dropped just as before. Note that the XDP program loaded alsoneeds to be in multi-buffer mode. This can be accomplished by using“xdp.frags” as the section name of the XDP program used.

To represent a packet consisting of multiple frames, a new flag calledXDP_PKT_CONTD is introduced in the options field of the Rx and Txdescriptors. If it is true (1) the packet continues with the nextdescriptor and if it is false (0) it means this is the last descriptorof the packet. Why the reverse logic of end-of-packet (eop) flag foundin many NICs? Just to preserve compatibility with non-multi-bufferapplications that have this bit set to false for all packets on Rx,and the apps set the options field to zero for Tx, as anything elsewill be treated as an invalid descriptor.

These are the semantics for producing packets onto AF_XDP Tx ringconsisting of multiple frames:

• When an invalid descriptor is found, all the otherdescriptors/frames of this packet are marked as invalid and notcompleted. The next descriptor is treated as the start of a newpacket, even if this was not the intent (because we cannot guessthe intent). As before, if your program is producing invaliddescriptors you have a bug that must be fixed.

• Zero length descriptors are treated as invalid descriptors.

• For copy mode, the maximum supported number of frames in a packet isequal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, alldescriptors accumulated so far are dropped and treated asinvalid. To produce an application that will work on any systemregardless of this config setting, limit the number of frags to 18,as the minimum value of the config is 17.

• For zero-copy mode, the limit is up to what the NIC HWsupports. Usually at least five on the NICs we have checked. Weconsciously chose to not enforce a rigid limit (such asCONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would haveresulted in copy actions under the hood to fit into what limit theNIC supports. Kind of defeats the purpose of zero-copy mode. How toprobe for this limit is explained in the “probe for multi-buffersupport” section.

On the Rx path in copy-mode, the xsk core copies the XDP data intomultiple descriptors, if needed, and sets the XDP_PKT_CONTD flag asdetailed before. Zero-copy mode works the same, though the data is notcopied. When the application gets a descriptor with the XDP_PKT_CONTDflag set to one, it means that the packet consists of multiple buffersand it continues with the next buffer in the followingdescriptor. When a descriptor with XDP_PKT_CONTD == 0 is received, itmeans that this is the last buffer of the packet. AF_XDP guaranteesthat only a complete packet (all frames in the packet) is sent to theapplication. If there is not enough space in the AF_XDP Rx ring, allframes of the packet will be dropped.

If application reads a batch of descriptors, using for example the libxdpinterfaces, it is not guaranteed that the batch will end with a fullpacket. It might end in the middle of a packet and the rest of thebuffers of that packet will arrive at the beginning of the next batch,since the libxdp interface does not read the whole ring (unless youhave an enormous batch size or a very small ring size).

An example program each for Rx and Tx multi-buffer support can be foundlater in this document.

SEC("xdp_sock")intxdp_sock_prog(structxdp_md*ctx){intindex=ctx->rx_queue_index;// A set entry here means that the corresponding queue_id// has an active AF_XDP socket bound to it.if(bpf_map_lookup_elem(&xsks_map,&index))returnbpf_redirect_map(&xsks_map,index,0);returnXDP_PASS;}

// struct xdp_rxtx_ring {//     __u32 *producer;//     __u32 *consumer;//     struct xdp_desc *desc;// };// struct xdp_umem_ring {//     __u32 *producer;//     __u32 *consumer;//     __u64 *desc;// };// typedef struct xdp_rxtx_ring RING;// typedef struct xdp_umem_ring RING;// typedef struct xdp_desc RING_TYPE;// typedef __u64 RING_TYPE;intdequeue_one(RING*ring,RING_TYPE*item){__u32entries=*ring->producer-*ring->consumer;if(entries==0)return-1;// read-barrier!*item=ring->desc[*ring->consumer&(RING_SIZE-1)];(*ring->consumer)++;return0;}intenqueue_one(RING*ring,constRING_TYPE*item){u32free_entries=RING_SIZE-(*ring->producer-*ring->consumer);if(free_entries==0)return-1;ring->desc[*ring->producer&(RING_SIZE-1)]=*item;// write-barrier!(*ring->producer)++;return0;}

voidrx_packets(structxsk_socket_info*xsk){staticboolnew_packet=true;u32idx_rx=0,idx_fq=0;staticchar*pkt;intrcvd=xsk_ring_cons__peek(&xsk->rx,opt_batch_size,&idx_rx);xsk_ring_prod__reserve(&xsk->umem->fq,rcvd,&idx_fq);for(inti=0;i<rcvd;i++){structxdp_desc*desc=xsk_ring_cons__rx_desc(&xsk->rx,idx_rx++);char*frag=xsk_umem__get_data(xsk->umem->buffer,desc->addr);booleop=!(desc->options&XDP_PKT_CONTD);if(new_packet)pkt=frag;elseadd_frag_to_pkt(pkt,frag);if(eop)process_pkt(pkt);new_packet=eop;*xsk_ring_prod__fill_addr(&xsk->umem->fq,idx_fq++)=desc->addr;}xsk_ring_prod__submit(&xsk->umem->fq,rcvd);xsk_ring_cons__release(&xsk->rx,rcvd);}

voidtx_packets(structxsk_socket_info*xsk,structpkt*pkts,intbatch_size){u32idx,i,pkt_nb=0;xsk_ring_prod__reserve(&xsk->tx,batch_size,&idx);for(i=0;i<batch_size;){u64addr=pkts[pkt_nb].addr;u32len=pkts[pkt_nb].size;do{structxdp_desc*tx_desc;tx_desc=xsk_ring_prod__tx_desc(&xsk->tx,idx+i++);tx_desc->addr=addr;if(len>xsk_frame_size){tx_desc->len=xsk_frame_size;tx_desc->options=XDP_PKT_CONTD;}else{tx_desc->len=len;tx_desc->options=0;pkt_nb++;}len-=tx_desc->len;addr+=xsk_frame_size;if(i==batch_size){/* Remember len, addr, pkt_nb for next iteration.                 * Skipped for simplicity.                 */break;}}while(len);}xsk_ring_prod__submit(&xsk->tx,i);}

Sample application¶

There is a xdpsock benchmarking/test application that can be found athttps://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-examplethat demonstrates how to use AF_XDP sockets with privateUMEMs. Say that you would like your UDP traffic from port 4242 to endup in queue 16, that we will enable AF_XDP on. Here, we use ethtoolfor this:

ethtool -N p3p2 rx-flow-hash udp4 fnethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \    action 16

Running the rxdrop benchmark in XDP_DRV mode can then be doneusing:

samples/bpf/xdpsock -i p3p2 -q 16 -r -N

For XDP_SKB mode, use the switch “-S” instead of “-N” and all optionscan be displayed with “-h”, as usual.

This sample application uses libbpf to make the setup and usage ofAF_XDP simpler. If you want to know how the raw uapi of AF_XDP isreally used to make something more advanced, take a look at the libbpfcode in tools/testing/selftests/bpf/xsk.[ch].

FAQ¶

Q: I am not seeing any traffic on the socket. What am I doing wrong?

A: When a netdev of a physical NIC is initialized, Linux usually

allocates one RX and TX queue pair per core. So on a 8 core system,queue ids 0 to 7 will be allocated, one per core. In the AF_XDPbind call or the xsk_socket__create libbpf function call, youspecify a specific queue id to bind to and it is only the traffictowards that queue you are going to get on you socket. So in theexample above, if you bind to queue 0, you are NOT going to get anytraffic that is distributed to queues 1 through 7. If you arelucky, you will see the traffic, but usually it will end up on oneof the queues you have not bound to.

There are a number of ways to solve the problem of getting thetraffic you want to the queue id you bound to. If you want to seeall the traffic, you can force the netdev to only have 1 queue, queueid 0, and then bind to queue 0. You can use ethtool to do this:

sudo ethtool -L <interface> combined 1

If you want to only see part of the traffic, you can program theNIC through ethtool to filter out your traffic to a single queue idthat you can bind your XDP socket to. Here is one example in whichUDP traffic to and from port 4242 are sent to queue 2:

sudo ethtool -N <interface> rx-flow-hash udp4 fnsudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \4242 action 2

A number of other ways are possible all up to the capabilities ofthe NIC you have.

Q: Can I use the XSKMAP to implement a switch between different umems

in copy mode?

A: The short answer is no, that is not supported at the moment. The

XSKMAP can only be used to switch traffic coming in on queue id Xto sockets bound to the same queue id X. The XSKMAP can containsockets bound to different queue ids, for example X and Y, but onlytraffic goming in from queue id Y can be directed to sockets boundto the same queue id Y. In zero-copy mode, you should use theswitch, or other distribution mechanism, in your NIC to directtraffic to the correct queue id and socket.

Q: My packets are sometimes corrupted. What is wrong?

A: Care has to be taken not to feed the same buffer in the UMEM into

more than one ring at the same time. If you for example feed thesame buffer into the FILL ring and the TX ring at the same time, theNIC might receive data into the buffer at the same time it issending it. This will cause some packets to become corrupted. Samething goes for feeding the same buffer into the FILL ringsbelonging to different queue ids or netdevs bound with theXDP_SHARED_UMEM flag.

Credits¶

• Björn Töpel (AF_XDP core)

• Magnus Karlsson (AF_XDP core)

• Alexander Duyck

• Alexei Starovoitov

• Daniel Borkmann

• Jesper Dangaard Brouer

• John Fastabend

• Jonathan Corbet (LWN coverage)

• Michael S. Tsirkin

• Qi Z Zhang

• Willem de Bruijn