1 .. SPDX-License-Identifier: GPL-2.0-only
2 .. include:: <isonum.txt>
8 :Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
9 :Author: Neo Jia <cjia@nvidia.com>
10 :Author: Kirti Wankhede <kwankhede@nvidia.com>
14 Virtual Function I/O (VFIO) Mediated devices[1]
15 ===============================================
17 The number of use cases for virtualizing DMA devices that do not have built-in
18 SR_IOV capability is increasing. Previously, to virtualize such devices,
19 developers had to create their own management interfaces and APIs, and then
20 integrate them with user space software. To simplify integration with user space
21 software, we have identified common requirements and a unified management
22 interface for such devices.
24 The VFIO driver framework provides unified APIs for direct device access. It is
25 an IOMMU/device-agnostic framework for exposing direct device access to user
26 space in a secure, IOMMU-protected environment. This framework is used for
27 multiple devices, such as GPUs, network adapters, and compute accelerators. With
28 direct device access, virtual machines or user space applications have direct
29 access to the physical device. This framework is reused for mediated devices.
31 The mediated core driver provides a common interface for mediated device
32 management that can be used by drivers of different devices. This module
33 provides a generic interface to perform these operations:
35 * Create and destroy a mediated device
36 * Add a mediated device to and remove it from a mediated bus driver
37 * Add a mediated device to and remove it from an IOMMU group
39 The mediated core driver also provides an interface to register a bus driver.
40 For example, the mediated VFIO mdev driver is designed for mediated devices and
41 supports VFIO APIs. The mediated bus driver adds a mediated device to and
42 removes it from a VFIO group.
44 The following high-level block diagram shows the main components and interfaces
45 in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
46 devices as examples, as these devices are the first devices to use this module::
50 | +-----------+ | mdev_register_driver() +--------------+
51 | | | +<------------------------+ |
53 | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user
54 | | driver | | probe()/remove() | | APIs
55 | | | | +--------------+
61 | +-----------+ | mdev_register_parent() +--------------+
62 | | | +<------------------------+ |
63 | | | | | nvidia.ko |<-> physical
64 | | | +------------------------>+ | device
65 | | | | callbacks +--------------+
67 | | device | | mdev_register_parent() +--------------+
68 | | interface | |<------------------------+ |
69 | | | | | i915.ko |<-> physical
70 | | | +------------------------>+ | device
71 | | | | callbacks +--------------+
73 | | | | mdev_register_parent() +--------------+
74 | | | +<------------------------+ |
75 | | | | | ccw_device.ko|<-> physical
76 | | | +------------------------>+ | device
77 | | | | callbacks +--------------+
82 Registration Interfaces
83 =======================
85 The mediated core driver provides the following types of registration
88 * Registration interface for a mediated bus driver
89 * Physical device driver interface
91 Registration Interface for a Mediated Bus Driver
92 ------------------------------------------------
94 The registration interface for a mediated device driver provides the following
95 structure to represent a mediated device's driver::
98 * struct mdev_driver [2] - Mediated device's driver
99 * @probe: called when new device created
100 * @remove: called when device removed
101 * @driver: device driver structure
104 int (*probe) (struct mdev_device *dev);
105 void (*remove) (struct mdev_device *dev);
106 const struct attribute * const *types_attrs;
107 struct device_driver driver;
110 A mediated bus driver for mdev should use this structure in the function calls
111 to register and unregister itself with the core driver:
115 int mdev_register_driver(struct mdev_driver *drv);
119 void mdev_unregister_driver(struct mdev_driver *drv);
121 The mediated bus driver's probe function should create a vfio_device on top of
122 the mdev_device and connect it to an appropriate implementation of
125 When a driver wants to add the GUID creation sysfs to an existing device it has
126 probe'd to then it should call::
128 int mdev_register_parent(struct mdev_parent *parent, struct device *dev,
129 struct mdev_driver *mdev_driver);
131 This will provide the 'mdev_supported_types/XX/create' files which can then be
132 used to trigger the creation of a mdev_device. The created mdev_device will be
133 attached to the specified driver.
135 When the driver needs to remove itself it calls::
137 void mdev_unregister_parent(struct mdev_parent *parent);
139 Which will unbind and destroy all the created mdevs and remove the sysfs files.
141 Mediated Device Management Interface Through sysfs
142 ==================================================
144 The management interface through sysfs enables user space software, such as
145 libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
146 This management interface provides flexibility to the underlying physical
147 device's driver to support features such as:
149 * Mediated device hot plug
150 * Multiple mediated devices in a single virtual machine
151 * Multiple mediated devices from different physical devices
153 Links in the mdev_bus Class Directory
154 -------------------------------------
155 The /sys/class/mdev_bus/ directory contains links to devices that are registered
156 with the mdev core driver.
158 Directories and files under the sysfs for Each Physical Device
159 --------------------------------------------------------------
163 |- [parent physical device]
164 |--- Vendor-specific-attributes [optional]
165 |--- [mdev_supported_types]
169 | | |--- available_instances
176 | | |--- available_instances
183 | |--- available_instances
188 * [mdev_supported_types]
190 The list of currently supported mediated device types and their details.
192 [<type-id>], device_api, and available_instances are mandatory attributes
193 that should be provided by vendor driver.
197 The [<type-id>] name is created by adding the device driver string as a prefix
198 to the string provided by the vendor driver. This format of this name is as
201 sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
203 (or using mdev_parent_dev(mdev) to arrive at the parent device outside
204 of the core mdev code)
208 This attribute should show which device API is being created, for example,
209 "vfio-pci" for a PCI device.
211 * available_instances
213 This attribute should show the number of devices of type <type-id> that can be
218 This directory contains links to the devices of type <type-id> that have been
223 This attribute should show human readable name. This is optional attribute.
227 This attribute should show brief features/description of the type. This is
230 Directories and Files Under the sysfs for Each mdev Device
231 ----------------------------------------------------------
235 |- [parent phy device]
238 |--- mdev_type {link to its type}
239 |--- vendor-specific-attributes [optional]
241 * remove (write only)
243 Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
244 fail the remove() callback if that device is active and the vendor driver
245 doesn't support hot unplug.
249 # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
251 Mediated device Hot plug
252 ------------------------
254 Mediated devices can be created and assigned at runtime. The procedure to hot
255 plug a mediated device is the same as the procedure to hot plug a PCI device.
257 Translation APIs for Mediated Devices
258 =====================================
260 The following APIs are provided for translating user pfn to host pfn in a VFIO
263 int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
264 int npage, int prot, struct page **pages);
266 void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova,
269 These functions call back into the back-end IOMMU module by using the pin_pages
270 and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
271 these callbacks are supported in the TYPE1 IOMMU module. To enable them for
272 other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
273 these two callback functions.
275 Using the Sample Code
276 =====================
278 mtty.c in samples/vfio-mdev/ directory is a sample driver program to
279 demonstrate how to use the mediated device framework.
281 The sample driver creates an mdev device that simulates a serial port over a PCI
284 1. Build and load the mtty.ko module.
286 This step creates a dummy device, /sys/devices/virtual/mtty/mtty/
288 Files in this device directory in sysfs are similar to the following::
290 # tree /sys/devices/virtual/mtty/mtty/
291 /sys/devices/virtual/mtty/mtty/
292 |-- mdev_supported_types
294 | | |-- available_instances
300 | |-- available_instances
306 | `-- sample_mtty_dev
308 | |-- autosuspend_delay_ms
310 | |-- runtime_active_time
312 | `-- runtime_suspended_time
313 |-- subsystem -> ../../../../class/mtty
316 2. Create a mediated device by using the dummy device that you created in the
319 # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \
320 /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
322 3. Add parameters to qemu-kvm::
325 sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
329 In the Linux guest VM, with no hardware on the host, the device appears
332 # lspci -s 00:05.0 -xxvv
333 00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
334 Subsystem: Device 4348:3253
336 Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
337 Stepping- SERR- FastB2B- DisINTx-
338 Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
339 <TAbort- <MAbort- >SERR- <PERR- INTx-
340 Interrupt: pin A routed to IRQ 10
341 Region 0: I/O ports at c150 [size=8]
342 Region 1: I/O ports at c158 [size=8]
343 Kernel driver in use: serial
344 00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
345 10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
346 20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
347 30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
349 In the Linux guest VM, dmesg output for the device is as follows:
351 serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
352 0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
353 0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A
356 5. In the Linux guest VM, check the serial ports::
358 # setserial -g /dev/ttyS*
359 /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
360 /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
361 /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
363 6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
364 /dev/ttyS2 with hardware flow control disabled.
366 7. Type data on the minicom terminal or send data to the terminal emulation
367 program and read the data.
369 Data is loop backed from hosts mtty driver.
371 8. Destroy the mediated device that you created::
373 # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
378 1. See Documentation/driver-api/vfio.rst for more information on VFIO.
379 2. struct mdev_driver in include/linux/mdev.h
380 3. struct mdev_parent_ops in include/linux/mdev.h
381 4. struct vfio_iommu_driver_ops in include/linux/vfio.h