The VFIO AP device driver¶

The VFIO AP (vfio_ap) device driver serves the following purposes:

1. Provides the interfaces to secure APQNs for exclusive use of KVM guests.

2. Sets up the VFIO mediated device interfaces to manage a vfio_ap mediateddevice and creates the sysfs interfaces for assigning adapters, usagedomains, and control domains comprising the matrix for a KVM guest.

3. Configures the APM, AQM and ADM in the APCB contained in the CRYCB referencedby a KVM guest’s SIE state description to grant the guest access to a matrixof AP devices

Reserve APQNs for exclusive use of KVM guests¶

The following block diagram illustrates the mechanism by which APQNs arereserved:

                              +------------------+               7 remove       |                  |         +--------------------> cex4queue driver |         |                    |                  |         |                    +------------------+         |         |         |                    +------------------+          +----------------+         |  5 register driver |                  | 3 create |                |         |   +---------------->   Device core    +---------->  matrix device |         |   |                |                  |          |                |         |   |                +--------^---------+          +----------------+         |   |                         |         |   |                         +-------------------+         |   | +-----------------------------------+       |         |   | |      4 register AP driver         |       | 2 register device         |   | |                                   |       |+--------+---+-v---+                      +--------+-------+-+|                  |                      |                  ||      ap_bus      +--------------------- >  vfio_ap driver  ||                  |       8 probe        |                  |+--------^---------+                      +--^--^------------+6 edit   |                                   |  |  apmask |     +-----------------------------+  | 11 mdev create  aqmask |     |           1 modprobe           |+--------+-----+---+           +----------------+-+         +----------------+|                  |           |                  |10 create|     mediated   ||      admin       |           | VFIO device core |--------->     matrix     ||                  +           |                  |         |     device     |+------+-+---------+           +--------^---------+         +--------^-------+       | |                              |                            |       | | 9 create vfio_ap-passthrough |                            |       | +------------------------------+                            |       +-------------------------------------------------------------+                   12  assign adapter/domain/control domain

The process for reserving an AP queue for use by a KVM guest is:

1. The administrator loads the vfio_ap device driver

2. The vfio-ap driver during its initialization will register a single ‘matrix’device with the device core. This will serve as the parent device forall vfio_ap mediated devices used to configure an AP matrix for a guest.

3. The /sys/devices/vfio_ap/matrix device is created by the device core

4. The vfio_ap device driver will register with the AP bus for AP queue devicesof type 10 and higher (CEX4 and newer). The driver will provide the vfio_apdriver’s probe and remove callback interfaces. Devices older than CEX4 queuesare not supported to simplify the implementation by not needlesslycomplicating the design by supporting older devices that will go out ofservice in the relatively near future, and for which there are few oldersystems around on which to test.

5. The AP bus registers the vfio_ap device driver with the device core

6. The administrator edits the AP adapter and queue masks to reserve AP queuesfor use by the vfio_ap device driver.

7. The AP bus removes the AP queues reserved for the vfio_ap driver from thedefault zcrypt cex4queue driver.

8. The AP bus probes the vfio_ap device driver to bind the queues reserved forit.

9. The administrator creates a passthrough type vfio_ap mediated device to beused by a guest

10. The administrator assigns the adapters, usage domains and control domainsto be exclusively used by a guest.

Set up the VFIO mediated device interfaces¶

The VFIO AP device driver utilizes the common interfaces of the VFIO mediateddevice core driver to:

• Register an AP mediated bus driver to add a vfio_ap mediated device to andremove it from a VFIO group.

• Create and destroy a vfio_ap mediated device

• Add a vfio_ap mediated device to and remove it from the AP mediated bus driver

• Add a vfio_ap mediated device to and remove it from an IOMMU group

The following high-level block diagram shows the main components and interfacesof the VFIO AP mediated device driver:

+-------------+|             || +---------+ | mdev_register_driver() +--------------+| |  Mdev   | +<-----------------------+              || |  bus    | |                        | vfio_mdev.ko || | driver  | +----------------------->+              |<-> VFIO user| +---------+ |    probe()/remove()    +--------------+    APIs|             ||  MDEV CORE  ||   MODULE    ||   mdev.ko   || +---------+ | mdev_register_parent() +--------------+| |Physical | +<-----------------------+              || | device  | |                        |  vfio_ap.ko  |<-> matrix| |interface| +----------------------->+              |    device| +---------+ |       callback         +--------------++-------------+

During initialization of the vfio_ap module, the matrix device is registeredwith an ‘mdev_parent_ops’ structure that provides the sysfs attributestructures, mdev functions and callback interfaces for managing the mediatedmatrix device.

• sysfs attribute structures:

  supported_type_groups

  The VFIO mediated device framework supports creation of user-definedmediated device types. These mediated device types are specifiedvia the ‘supported_type_groups’ structure when a device is registeredwith the mediated device framework. The registration process creates thesysfs structures for each mediated device type specified in the‘mdev_supported_types’ sub-directory of the device being registered. Alongwith the device type, the sysfs attributes of the mediated device type areprovided.

  The VFIO AP device driver will register one mediated device type forpassthrough devices:

  /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough

  Only the read-only attributes required by the VFIO mdev framework willbe provided:

  ... name... device_api... available_instances... device_api

  Where:

  • name:

    specifies the name of the mediated device type

  • device_api:

    the mediated device type’s API

  • available_instances:

    the number of vfio_ap mediated passthrough devicesthat can be created

  • device_api:

    specifies the VFIO API

  mdev_attr_groups

  This attribute group identifies the user-defined sysfs attributes of themediated device. When a device is registered with the VFIO mediated deviceframework, the sysfs attribute files identified in the ‘mdev_attr_groups’structure will be created in the vfio_ap mediated device’s directory. Thesysfs attributes for a vfio_ap mediated device are:

  assign_adapter / unassign_adapter:

  Write-only attributes for assigning/unassigning an AP adapter to/from thevfio_ap mediated device. To assign/unassign an adapter, the APID of theadapter is echoed into the respective attribute file.

  assign_domain / unassign_domain:

  Write-only attributes for assigning/unassigning an AP usage domain to/fromthe vfio_ap mediated device. To assign/unassign a domain, the domainnumber of the usage domain is echoed into the respective attributefile.

  matrix:

  A read-only file for displaying the APQNs derived from the Cartesianproduct of the adapter and domain numbers assigned to the vfio_ap mediateddevice.

  guest_matrix:

  A read-only file for displaying the APQNs derived from the Cartesianproduct of the adapter and domain numbers assigned to the APM and AQMfields respectively of the KVM guest’s CRYCB. This may differ from thethe APQNs assigned to the vfio_ap mediated device if any APQN does notreference a queue device bound to the vfio_ap device driver (i.e., thequeue is not in the host’s AP configuration).

  assign_control_domain / unassign_control_domain:

  Write-only attributes for assigning/unassigning an AP control domainto/from the vfio_ap mediated device. To assign/unassign a control domain,the ID of the domain to be assigned/unassigned is echoed into therespective attribute file.

  control_domains:

  A read-only file for displaying the control domain numbers assigned to thevfio_ap mediated device.

  ap_config:

  A read/write file that, when written to, allows all three of thevfio_ap mediated device’s ap matrix masks to be replaced in one shot.Three masks are given, one for adapters, one for domains, and one forcontrol domains. If the given state cannot be set then no changes aremade to the vfio-ap mediated device.

  The format of the data written to ap_config is as follows:{amask},{dmask},{cmask}n

  n is a newline character.

  amask, dmask, and cmask are masks identifying which adapters, domains,and control domains should be assigned to the mediated device.

  The format of a mask is as follows:0xNN..NN

  Where NN..NN is 64 hexadecimal characters representing a 256-bit value.The leftmost (highest order) bit represents adapter/domain 0.

  For an example set of masks that represent your mdev’s currentconfiguration, simply cat ap_config.

  Setting an adapter or domain number greater than the maximum allowed forthe system will result in an error.

  This attribute is intended to be used by automation. End users would bebetter served using the respective assign/unassign attributes foradapters, domains, and control domains.

• functions:

  create:

  allocates the ap_matrix_mdev structure used by the vfio_ap driver to:

  • Store the reference to the KVM structure for the guest using the mdev

  • Store the AP matrix configuration for the adapters, domains, and controldomains assigned via the corresponding sysfs attributes files

  • Store the AP matrix configuration for the adapters, domains and controldomains available to a guest. A guest may not be provided access to APQNsreferencing queue devices that do not exist, or are not bound to thevfio_ap device driver.

  remove:

  deallocates the vfio_ap mediated device’s ap_matrix_mdev structure.This will be allowed only if a running guest is not using the mdev.

• callback interfaces

  open_device:

  The vfio_ap driver uses this callback to register aVFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the matrix mdevdevices. The open_device callback is invoked by userspace to connect theVFIO iommu group for the matrix mdev device to the MDEV bus. Access to theKVM structure used to configure the KVM guest is provided via this callback.The KVM structure, is used to configure the guest’s access to the AP matrixdefined via the vfio_ap mediated device’s sysfs attribute files.

  close_device:

  unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for thematrix mdev device and deconfigures the guest’s AP matrix.

  ioctl:

  this callback handles the VFIO_DEVICE_GET_INFO and VFIO_DEVICE_RESET ioctlsdefined by the vfio framework.

Configure the guest’s AP resources¶

Configuring the AP resources for a KVM guest will be performed when theVFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifierfunction is called when userspace connects to KVM. The guest’s AP resources areconfigured via its APCB by:

• Setting the bits in the APM corresponding to the APIDs assigned to thevfio_ap mediated device via its ‘assign_adapter’ interface.

• Setting the bits in the AQM corresponding to the domains assigned to thevfio_ap mediated device via its ‘assign_domain’ interface.

• Setting the bits in the ADM corresponding to the domain dIDs assigned to thevfio_ap mediated device via its ‘assign_control_domains’ interface.

The linux device model precludes passing a device through to a KVM guest thatis not bound to the device driver facilitating its pass-through. Consequently,an APQN that does not reference a queue device bound to the vfio_ap devicedriver will not be assigned to a KVM guest’s matrix. The AP architecture,however, does not provide a means to filter individual APQNs from the guest’smatrix, so the adapters, domains and control domains assigned to vfio_apmediated device via its sysfs ‘assign_adapter’, ‘assign_domain’ and‘assign_control_domain’ interfaces will be filtered before providing the APconfiguration to a guest:

• The APIDs of the adapters, the APQIs of the domains and the domain numbers ofthe control domains assigned to the matrix mdev that are not also assigned tothe host’s AP configuration will be filtered.

• Each APQN derived from the Cartesian product of the APIDs and APQIs assignedto the vfio_ap mdev is examined and if any one of them does not reference aqueue device bound to the vfio_ap device driver, the adapter will not beplugged into the guest (i.e., the bit corresponding to its APID will not beset in the APM of the guest’s APCB).

The CPU model features for AP¶

The AP stack relies on the presence of the AP instructions as well as threefacilities: The AP Facilities Test (APFT) facility; the AP QueryConfiguration Information (QCI) facility; and the AP Queue Interruption Controlfacility. These features/facilities are made available to a KVM guest via thefollowing CPU model features:

1. ap: Indicates whether the AP instructions are installed on the guest. Thisfeature will be enabled by KVM only if the AP instructions are installedon the host.

2. apft: Indicates the APFT facility is available on the guest. This facilitycan be made available to the guest only if it is available on the host (i.e.,facility bit 15 is set).

3. apqci: Indicates the AP QCI facility is available on the guest. This facilitycan be made available to the guest only if it is available on the host (i.e.,facility bit 12 is set).

4. apqi: Indicates AP Queue Interruption Control faclity is available on theguest. This facility can be made available to the guest only if it isavailable on the host (i.e., facility bit 65 is set).

Note: If the user chooses to specify a CPU model different than the ‘host’model to QEMU, the CPU model features and facilities need to be turned onexplicitly; for example:

/usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on

A guest can be precluded from using AP features/facilities by turning them offexplicitly; for example:

/usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off

Note: If the APFT facility is turned off (apft=off) for the guest, the guestwill not see any AP devices. The zcrypt device drivers on the guest thatregister for type 10 and newer AP devices - i.e., the cex4card and cex4queuedevice drivers - need the APFT facility to ascertain the facilities installed ona given AP device. If the APFT facility is not installed on the guest, then noadapter or domain devices will get created by the AP bus running on theguest because only type 10 and newer devices can be configured for guest use.

Guest1¶

Guest2¶

Guest3¶

These are the steps:

1. Install the vfio_ap module on the linux host. The dependency chain for thevfio_ap module is:* iommu* s390* zcrypt* vfio* vfio_mdev* vfio_mdev_device* KVM

   To build the vfio_ap module, the kernel build must be configured with thefollowing Kconfig elements selected:* IOMMU_SUPPORT* S390* AP* VFIO* KVM

   If using make menuconfig select the following to build the vfio_ap module:

   -> Device Drivers   -> IOMMU Hardware Support      select S390 AP IOMMU Support   -> VFIO Non-Privileged userspace driver framework      -> Mediated device driver frramework         -> VFIO driver for Mediated devices-> I/O subsystem   -> VFIO support for AP devices

2. Secure the AP queues to be used by the three guests so that the host can notaccess them. To secure them, there are two sysfs files that specifybitmasks marking a subset of the APQN range as usable only by the default APqueue device drivers. All remaining APQNs are available for use byany other device driver. The vfio_ap device driver is currently the onlynon-default device driver. The location of the sysfs files containing themasks are:

   /sys/bus/ap/apmask/sys/bus/ap/aqmask

   The ‘apmask’ is a 256-bit mask that identifies a set of AP adapter IDs(APID). Each bit in the mask, from left to right, corresponds to an APID from0-255. If a bit is set, the APID belongs to the subset of APQNs marked asavailable only to the default AP queue device drivers.

   The ‘aqmask’ is a 256-bit mask that identifies a set of AP queue indexes(APQI). Each bit in the mask, from left to right, corresponds to an APQI from0-255. If a bit is set, the APQI belongs to the subset of APQNs marked asavailable only to the default AP queue device drivers.

   The Cartesian product of the APIDs corresponding to the bits set in theapmask and the APQIs corresponding to the bits set in the aqmask comprisethe subset of APQNs that can be used only by the host default device drivers.All other APQNs are available to the non-default device drivers such as thevfio_ap driver.

   Take, for example, the following masks:

   apmask:0x7d00000000000000000000000000000000000000000000000000000000000000aqmask:0x8000000000000000000000000000000000000000000000000000000000000000

   The masks indicate:

   • Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host defaultdevice drivers.

   • Domain 0 is available for use by the host default device drivers

   • The subset of APQNs available for use only by the default host devicedrivers are:

     (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0)

   • All other APQNs are available for use by the non-default device drivers.

   The APQN of each AP queue device assigned to the linux host is checked by theAP bus against the set of APQNs derived from the Cartesian product of APIDsand APQIs marked as available to the default AP queue device drivers. If amatch is detected, only the default AP queue device drivers will be probed;otherwise, the vfio_ap device driver will be probed.

   By default, the two masks are set to reserve all APQNs for use by the defaultAP queue device drivers. There are two ways the default masks can be changed:

   1. The sysfs mask files can be edited by echoing a string into therespective sysfs mask file in one of two formats:

      • An absolute hex string starting with 0x - like “0x12345678” - setsthe mask. If the given string is shorter than the mask, it is paddedwith 0s on the right; for example, specifying a mask value of 0x41 isthe same as specifying:

        0x4100000000000000000000000000000000000000000000000000000000000000

        Keep in mind that the mask reads from left to right, so the maskabove identifies device numbers 1 and 7 (01000001).

        If the string is longer than the mask, the operation is terminated withan error (EINVAL).

      • Individual bits in the mask can be switched on and off by specifyingeach bit number to be switched in a comma separated list. Each bitnumber string must be prepended with a (‘+’) or minus (‘-’) to indicatethe corresponding bit is to be switched on (‘+’) or off (‘-‘). Somevalid values are:

        • “+0” switches bit 0 on

        • “-13” switches bit 13 off

        • “+0x41” switches bit 65 on

        • “-0xff” switches bit 255 off

        The following example:

        +0,-6,+0x47,-0xf0

        Switches bits 0 and 71 (0x47) on

        Switches bits 6 and 240 (0xf0) off

        Note that the bits not specified in the list remain as they were beforethe operation.

   2. The masks can also be changed at boot time via parameters on the kernelcommand line like this:

      ap.apmask=0xffff ap.aqmask=0x40

      This would create the following masks:

      apmask:0xffff000000000000000000000000000000000000000000000000000000000000aqmask:0x4000000000000000000000000000000000000000000000000000000000000000

      Resulting in these two pools:

      default drivers pool:    adapter 0-15, domain 1alternate drivers pool:  adapter 16-255, domains 0, 2-255

   Note:Changing a mask such that one or more APQNs will be taken from a vfio_apmediated device (see below) will fail with an error (EBUSY). A messageis logged to the kernel ring buffer which can be viewed with the ‘dmesg’command. The output identifies each APQN flagged as ‘in use’ and identifiesthe vfio_ap mediated device to which it is assigned; for example:

   Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4

Securing the APQNs for our example¶

To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,06.00ab, and 06.00ff for use by the vfio_ap device driver, the correspondingAPQNs can be removed from the default masks using either of the followingcommands:

echo -5,-6 > /sys/bus/ap/apmaskecho -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask

Or the masks can be set as follows:

echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \> apmaskecho 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \> aqmask

This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. Thesysfs directory for the vfio_ap device driver will now contain symbolic linksto the AP queue devices bound to it:

/sys/bus/ap... [drivers]...... [vfio_ap]......... [05.0004]......... [05.0047]......... [05.00ab]......... [05.00ff]......... [06.0004]......... [06.0047]......... [06.00ab]......... [06.00ff]

Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)can be bound to the vfio_ap device driver. The reason for this is tosimplify the implementation by not needlessly complicating the design bysupporting older devices that will go out of service in the relatively nearfuture and for which there are few older systems on which to test.

The administrator, therefore, must take care to secure only AP queues thatcan be bound to the vfio_ap device driver. The device type for a given APqueue device can be read from the parent card’s sysfs directory. For example,to see the hardware type of the queue 05.0004:

cat /sys/bus/ap/devices/card05/hwtype

The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to thevfio_ap device driver.

3. Create the mediated devices needed to configure the AP matrixes for thethree guests and to provide an interface to the vfio_ap driver foruse by the guests:

   /sys/devices/vfio_ap/matrix/--- [mdev_supported_types]------ [vfio_ap-passthrough] (passthrough vfio_ap mediated device type)--------- create--------- [devices]

   To create the mediated devices for the three guests:

   uuidgen > createuuidgen > createuuidgen > createorecho $uuid1 > createecho $uuid2 > createecho $uuid3 > create

   This will create three mediated devices in the [devices] subdirectory namedafter the UUID written to the create attribute file. We call them $uuid1,$uuid2 and $uuid3 and this is the sysfs directory structure after creation:

   /sys/devices/vfio_ap/matrix/--- [mdev_supported_types]------ [vfio_ap-passthrough]--------- [devices]------------ [$uuid1]--------------- assign_adapter--------------- assign_control_domain--------------- assign_domain--------------- matrix--------------- unassign_adapter--------------- unassign_control_domain--------------- unassign_domain------------ [$uuid2]--------------- assign_adapter--------------- assign_control_domain--------------- assign_domain--------------- matrix--------------- unassign_adapter----------------unassign_control_domain----------------unassign_domain------------ [$uuid3]--------------- assign_adapter--------------- assign_control_domain--------------- assign_domain--------------- matrix--------------- unassign_adapter----------------unassign_control_domain----------------unassign_domain

   Note*: The vfio_ap mdevs do not persist across reboots unless the

   mdevctl tool is used to create and persist them.

4. The administrator now needs to configure the matrixes for the mediateddevices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).

   This is how the matrix is configured for Guest1:

   echo 5 > assign_adapterecho 6 > assign_adapterecho 4 > assign_domainecho 0xab > assign_domain

   Control domains can similarly be assigned using the assign_control_domainsysfs file.

   If a mistake is made configuring an adapter, domain or control domain,you can use the unassign_xxx files to unassign the adapter, domain orcontrol domain.

   To display the matrix configuration for Guest1:

   cat matrix

   To display the matrix that is or will be assigned to Guest1:

   cat guest_matrix

   This is how the matrix is configured for Guest2:

   echo 5 > assign_adapterecho 0x47 > assign_domainecho 0xff > assign_domain

   This is how the matrix is configured for Guest3:

   echo 6 > assign_adapterecho 0x47 > assign_domainecho 0xff > assign_domain

   In order to successfully assign an adapter:

   • The adapter number specified must represent a value from 0 up to themaximum adapter number configured for the system. If an adapter numberhigher than the maximum is specified, the operation will terminate withan error (ENODEV).

     Note: The maximum adapter number can be obtained via the sysfs

     /sys/bus/ap/ap_max_adapter_id attribute file.

   • Each APQN derived from the Cartesian product of the APID of the adapterbeing assigned and the APQIs of the domains previously assigned:

     • Must only be available to the vfio_ap device driver as specified in thesysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If evenone APQN is reserved for use by the host device driver, the operationwill terminate with an error (EADDRNOTAVAIL).

     • Must NOT be assigned to another vfio_ap mediated device. If even one APQNis assigned to another vfio_ap mediated device, the operation willterminate with an error (EBUSY).

     • Must NOT be assigned while the sysfs /sys/bus/ap/apmask andsys/bus/ap/aqmask attribute files are being edited or the operation mayterminate with an error (EBUSY).

   In order to successfully assign a domain:

   • The domain number specified must represent a value from 0 up to themaximum domain number configured for the system. If a domain numberhigher than the maximum is specified, the operation will terminate withan error (ENODEV).

     Note: The maximum domain number can be obtained via the sysfs

     /sys/bus/ap/ap_max_domain_id attribute file.

   • Each APQN derived from the Cartesian product of the APQI of the domainbeing assigned and the APIDs of the adapters previously assigned:

   • Must only be available to the vfio_ap device driver as specified in thesysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If evenone APQN is reserved for use by the host device driver, the operationwill terminate with an error (EADDRNOTAVAIL).

   • Must NOT be assigned to another vfio_ap mediated device. If even one APQNis assigned to another vfio_ap mediated device, the operation willterminate with an error (EBUSY).

   • Must NOT be assigned while the sysfs /sys/bus/ap/apmask andsys/bus/ap/aqmask attribute files are being edited or the operation mayterminate with an error (EBUSY).

   In order to successfully assign a control domain:

   • The domain number specified must represent a value from 0 up to the maximumdomain number configured for the system. If a control domain number higherthan the maximum is specified, the operation will terminate with anerror (ENODEV).

5. Start Guest1:

   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \   -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...

7. Start Guest2:

   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \   -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...

7. Start Guest3:

   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \   -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...

When the guest is shut down, the vfio_ap mediated devices may be removed.

Using our example again, to remove the vfio_ap mediated device $uuid1:

/sys/devices/vfio_ap/matrix/   --- [mdev_supported_types]   ------ [vfio_ap-passthrough]   --------- [devices]   ------------ [$uuid1]   --------------- remove

echo 1 > remove

This will remove all of the matrix mdev device’s sysfs structures includingthe mdev device itself. To recreate and reconfigure the matrix mdev device,all of the steps starting with step 3 will have to be performed again. Notethat the remove will fail if a guest using the vfio_ap mdev is still running.

It is not necessary to remove a vfio_ap mdev, but one may want toremove it if no guest will use it during the remaining lifetime of the linuxhost. If the vfio_ap mdev is removed, one may want to also reconfigurethe pool of adapters and queues reserved for use by the default drivers.