Virtual Function I/O (VFIO) Mediated devices[1]¶

The number of use cases for virtualizing DMA devices that do not have built-inSR_IOV capability is increasing. Previously, to virtualize such devices,developers had to create their own management interfaces and APIs, and thenintegrate them with user space software. To simplify integration with user spacesoftware, we have identified common requirements and a unified managementinterface for such devices.

The VFIO driver framework provides unified APIs for direct device access. It isan IOMMU/device-agnostic framework for exposing direct device access to userspace in a secure, IOMMU-protected environment. This framework is used formultiple devices, such as GPUs, network adapters, and compute accelerators. Withdirect device access, virtual machines or user space applications have directaccess to the physical device. This framework is reused for mediated devices.

The mediated core driver provides a common interface for mediated devicemanagement that can be used by drivers of different devices. This moduleprovides a generic interface to perform these operations:

• Create and destroy a mediated device

• Add a mediated device to and remove it from a mediated bus driver

• Add a mediated device to and remove it from an IOMMU group

The mediated core driver also provides an interface to register a bus driver.For example, the mediated VFIO mdev driver is designed for mediated devices andsupports VFIO APIs. The mediated bus driver adds a mediated device to andremoves it from a VFIO group.

The following high-level block diagram shows the main components and interfacesin the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBMdevices as examples, as these devices are the first devices to use this module:

+---------------+|               || +-----------+ |  mdev_register_driver() +--------------+| |           | +<------------------------+              || |  mdev     | |                         |              || |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user| |  driver   | |     probe()/remove()    |              |    APIs| |           | |                         +--------------+| +-----------+ ||               ||  MDEV CORE    ||   MODULE      ||   mdev.ko     || +-----------+ |  mdev_register_parent() +--------------+| |           | +<------------------------+              || |           | |                         | ccw_device.ko|<-> physical| |           | +------------------------>+              |    device| |           | |        callbacks        +--------------+| | Physical  | || |  device   | |  mdev_register_parent() +--------------+| | interface | |<------------------------+              || |           | |                         |  i915.ko     |<-> physical| |           | +------------------------>+              |    device| |           | |        callbacks        +--------------+| +-----------+ |+---------------+

Registration Interfaces¶

The mediated core driver provides the following types of registrationinterfaces:

• Registration interface for a mediated bus driver

• Physical device driver interface

/* * struct mdev_driver [2] - Mediated device's driver * @probe: called when new device created * @remove: called when device removed * @driver: device driver structure */struct mdev_driver {        int  (*probe)  (struct mdev_device *dev);        void (*remove) (struct mdev_device *dev);        unsigned int (*get_available)(struct mdev_type *mtype);        ssize_t (*show_description)(struct mdev_type *mtype, char *buf);        struct device_driver    driver;};

int mdev_register_parent(struct mdev_parent *parent, struct device *dev,                    struct mdev_driver *mdev_driver);

void mdev_unregister_parent(struct mdev_parent *parent);

Mediated Device Management Interface Through sysfs¶

The management interface through sysfs enables user space software, such aslibvirt, to query and configure mediated devices in a hardware-agnostic fashion.This management interface provides flexibility to the underlying physicaldevice’s driver to support features such as:

• Mediated device hot plug

• Multiple mediated devices in a single virtual machine

• Multiple mediated devices from different physical devices

|- [parent physical device]|--- Vendor-specific-attributes [optional]|--- [mdev_supported_types]|     |--- [<type-id>]|     |   |--- create|     |   |--- name|     |   |--- available_instances|     |   |--- device_api|     |   |--- description|     |   |--- [devices]|     |--- [<type-id>]|     |   |--- create|     |   |--- name|     |   |--- available_instances|     |   |--- device_api|     |   |--- description|     |   |--- [devices]|     |--- [<type-id>]|          |--- create|          |--- name|          |--- available_instances|          |--- device_api|          |--- description|          |--- [devices]

|- [parent phy device]|--- [$MDEV_UUID]       |--- remove       |--- mdev_type {link to its type}       |--- vendor-specific-attributes [optional]

# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove

Translation APIs for Mediated Devices¶

The following APIs are provided for translating user pfn to host pfn in a VFIOdriver:

int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,                          int npage, int prot, struct page **pages);void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova,                            int npage);

These functions call back into the back-end IOMMU module by using the pin_pagesand unpin_pages callbacks of thestructvfio_iommu_driver_ops[4]. Currentlythese callbacks are supported in the TYPE1 IOMMU module. To enable them forother IOMMU backend modules, such as PPC64 sPAPR module, they need to providethese two callback functions.

References¶

1. SeeVFIO - “Virtual Function I/O” for more information on VFIO.

2. structmdev_driver in include/linux/mdev.h

3. structmdev_parent_ops in include/linux/mdev.h

4. structvfio_iommu_driver_ops in include/linux/vfio.h