Create/Destroy VDUSE devices¶

VDUSE devices are created as follows:

1. Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on/dev/vduse/control.

2. Setup each virtqueue with ioctl(VDUSE_VQ_SETUP) on /dev/vduse/$NAME.

3. Begin processing VDUSE messages from /dev/vduse/$NAME. The firstmessages will arrive while attaching the VDUSE instance to vDPA bus.

4. Send the VDPA_CMD_DEV_NEW netlink message to attach the VDUSEinstance to vDPA bus.

VDUSE devices are destroyed as follows:

1. Send the VDPA_CMD_DEV_DEL netlink message to detach the VDUSEinstance from vDPA bus.

2. Close the file descriptor referring to /dev/vduse/$NAME.

3. Destroy the VDUSE instance with ioctl(VDUSE_DESTROY_DEV) on/dev/vduse/control.

The netlink messages can be sent via vdpa tool in iproute2 or use thebelow sample codes:

staticintnetlink_add_vduse(constchar*name,enumvdpa_commandcmd){structnl_sock*nlsock;structnl_msg*msg;intfamid;nlsock=nl_socket_alloc();if(!nlsock)return-ENOMEM;if(genl_connect(nlsock))gotofree_sock;famid=genl_ctrl_resolve(nlsock,VDPA_GENL_NAME);if(famid<0)gotoclose_sock;msg=nlmsg_alloc();if(!msg)gotoclose_sock;if(!genlmsg_put(msg,NL_AUTO_PORT,NL_AUTO_SEQ,famid,0,0,cmd,0))gotonla_put_failure;NLA_PUT_STRING(msg,VDPA_ATTR_DEV_NAME,name);if(cmd==VDPA_CMD_DEV_NEW)NLA_PUT_STRING(msg,VDPA_ATTR_MGMTDEV_DEV_NAME,"vduse");if(nl_send_sync(nlsock,msg))gotoclose_sock;nl_close(nlsock);nl_socket_free(nlsock);return0;nla_put_failure:nlmsg_free(msg);close_sock:nl_close(nlsock);free_sock:nl_socket_free(nlsock);return-1;}

How VDUSE works¶

As mentioned above, a VDUSE device is created by ioctl(VDUSE_CREATE_DEV) on/dev/vduse/control. With this ioctl, userspace can specify some basic configurationsuch as device name (uniquely identify a VDUSE device), virtio features, virtioconfiguration space, the number of virtqueues and so on for this emulated device.Then a char device interface (/dev/vduse/$NAME) is exported to userspace for deviceemulation. Userspace can use the VDUSE_VQ_SETUP ioctl on /dev/vduse/$NAME toadd per-virtqueue configuration such as the max size of virtqueue to the device.

After the initialization, the VDUSE device can be attached to vDPA bus viathe VDPA_CMD_DEV_NEW netlink message. Userspace needs to read()/write() on/dev/vduse/$NAME to receive/reply some control messages from/to VDUSE kernelmodule as follows:

staticintvduse_message_handler(intdev_fd){intlen;structvduse_dev_requestreq;structvduse_dev_responseresp;len=read(dev_fd,&req,sizeof(req));if(len!=sizeof(req))return-1;resp.request_id=req.request_id;switch(req.type){/* handle different types of messages */}len=write(dev_fd,&resp,sizeof(resp));if(len!=sizeof(resp))return-1;return0;}

There are now three types of messages introduced by VDUSE framework:

• VDUSE_GET_VQ_STATE: Get the state for virtqueue, userspace should returnavail index for split virtqueue or the device/driver ring wrap counters andthe avail and used index for packed virtqueue.

• VDUSE_SET_STATUS: Set the device status, userspace should followthe virtio spec:https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.htmlto process this message. For example, fail to set the FEATURES_OK devicestatus bit if the device can not accept the negotiated virtio featuresget from the VDUSE_DEV_GET_FEATURES ioctl.

• VDUSE_UPDATE_IOTLB: Notify userspace to update the memory mapping for specifiedIOVA range, userspace should firstly remove the old mapping, then setup the newmapping via the VDUSE_IOTLB_GET_FD ioctl.

After DRIVER_OK status bit is set via the VDUSE_SET_STATUS message, userspace isable to start the dataplane processing as follows:

1. Get the specified virtqueue’s information with the VDUSE_VQ_GET_INFO ioctl,including the size, the IOVAs of descriptor table, available ring and used ring,the state and the ready status.

2. Pass the above IOVAs to the VDUSE_IOTLB_GET_FD ioctl so that those IOVA regionscan be mapped into userspace. Some sample codes is shown below:

staticintperm_to_prot(uint8_tperm){intprot=0;switch(perm){caseVDUSE_ACCESS_WO:prot|=PROT_WRITE;break;caseVDUSE_ACCESS_RO:prot|=PROT_READ;break;caseVDUSE_ACCESS_RW:prot|=PROT_READ|PROT_WRITE;break;}returnprot;}staticvoid*iova_to_va(intdev_fd,uint64_tiova,uint64_t*len){intfd;void*addr;size_tsize;structvduse_iotlb_entryentry;entry.start=iova;entry.last=iova;/*         * Find the first IOVA region that overlaps with the specified         * range [start, last] and return the corresponding file descriptor.         */fd=ioctl(dev_fd,VDUSE_IOTLB_GET_FD,&entry);if(fd<0)returnNULL;size=entry.last-entry.start+1;*len=entry.last-iova+1;addr=mmap(0,size,perm_to_prot(entry.perm),MAP_SHARED,fd,entry.offset);close(fd);if(addr==MAP_FAILED)returnNULL;/*         * Using some data structures such as linked list to store         * the iotlb mapping. The munmap(2) should be called for the         * cached mapping when the corresponding VDUSE_UPDATE_IOTLB         * message is received or the device is reset.         */returnaddr+iova-entry.start;}

3. Setup the kick eventfd for the specified virtqueues with the VDUSE_VQ_SETUP_KICKFDioctl. The kick eventfd is used by VDUSE kernel module to notify userspace toconsume the available ring. This is optional since userspace can choose to poll theavailable ring instead.

4. Listen to the kick eventfd (optional) and consume the available ring. The bufferdescribed by the descriptors in the descriptor table should be also mapped intouserspace via the VDUSE_IOTLB_GET_FD ioctl before accessing.

5. Inject an interrupt for specific virtqueue with the VDUSE_INJECT_VQ_IRQ ioctlafter the used ring is filled.

For more details on the uAPI, please see include/uapi/linux/vduse.h.