KVM, QEMU, and more

VIRTIO 1.2 is out!

2022-07-18T09:45:00+02:00

A new version of the virtio specification has been released! As it has been three years after the 1.1 release, quite a lot of changes have accumulated. I have attempted to list some of them below; for details, you are invited to check out the spec :)

There are already some changes queued for 1.3; let’s hope it won’t take us three years again before the next release ;)

New device types in 1.2

Several new device types have been added.

virtio-pmem: persistent memory device; useful to avoid a separate page cache in the guest
virtio-fs: access a file system; kind of the spritual successor to the never officially standardized virtio-9p
virtio-rpmb: a tamper-resistant and anti-replay storage device
virtio-iommu: can both be a proxy for a physical IOMMU, or act as a virtual IOMMU
virtio-snd: a sound card supporting input and output PCM streams
virtio-mem: provides a memory region in guest physical address space; useful to implement memory hot(un)plugging
virtio-i2c: a virtual I2C adapter
virtio-scmi: implements the Arm System Control and Management Interface, for things like sensors etc.
virtio-gpio: a virtual GPIO device to manage named I/O lines

New features for existing device types

Enhancements have been added to some already existing device types.

virtio-blk
- multiqueue support
- lifetime metrics
- secure erase
virtio-net
- support for the guest providing the exact header length
- receive-side scaling
- per-packet hash reporting
- per-virtqueue driver notifications
- UDP segmentation offload
virtio-gpu
- support for 3D commands
- resource sharing
- blob resources
- context initialization
virtio-balloon
- free page hints
- page poisoning
- free page reporting
virtio-vsock
- seqpacket sockets

Features not specific to a device type

Some general enhancements include:

support for vendor-specific PCI capabilities
support for sharing resources between devices
support for resetting individual virtqueues

QEMU machine types and compatibility (part 2)

2022-01-21T13:45:00+01:00

In the first part of this article, I talked about how you can use versioned machine types to ensure compatibility. But the more interesting part is how this actually works under the covers.

Device properties, and making them compatible

QEMU devices often come with a list of properties that influence how the device is created and how it operates. Typically, authors try to come up with reasonable default values, which may be overriden if desired. However, the idea of what is considered reasonable may change over time, and a newer QEMU may provide a different default value for a property.

If you want to migrate a guest from an older QEMU machine to a more recent QEMU, you obviously need to use the default values from that older QEMU machine as well. For that, QEMU uses arrays of GlobalPropery structures.

If you take a look at hw/core/machine.c, you will notice several arrays named hw_compat_<major>_<minor>. These contain triplets specifying (from right to left) the default value for a certain property for a certain device. The arrays are designed to be included by the compat machine for <major>.<minor>, thus specifying a default value for that machine version and older. (More on this later in this article.)

For example, QEMU 5.2 changed the default number of virtio queues defined for virtio-blk and virtio-scsi devices: prior to 5.1, one queue would be present if no other value had been specified; with 5.2, the default number of queues would align with the number of vcpus for virtio-pci. Therefore, hw_compat_5_1 contains the following lines:

{ "virtio-blk-device", "num-queues", "1"},
{ "virtio-scsi-device", "num_queues", "1"},

(and some corresponding lines for vhost.) This makes sure that any virtio-blk or virtio-scsi device on a -5.1 or older machine type will have one virtio queue per default. Note that this holds true for all virtio-blk and virtio-scsi devices, regardless of which transport they are using; for transports like ccw where nothing changed with 5.2, this simply does not make any difference.

Generally, statements for all devices can go into the hw_compat_ arrays; if a device is not present or even not available at all for the machine that is started, the statement will simply not take any effect.

x86 considerations

For the x86 machine types (pc-i440fx and pc-q35), pc_compat_<major>_<minor> arrays are defined in hw/i386/pc.c, mostly covering properties for x86 cpus, but also some other x86-specific devices.

Per-machine changes

Some incompatible changes are not happening at the device property level, so the compat properties approach cannot be used. Instead, the individual machines need to take care of those changes.

For example, in QEMU 6.2 the smp parsing code started to prefer cores over sockets instead of preferring sockets. Therefore, all 6.1 compat machines have code like

m->smp_props.prefer_sockets = true;

to set prefer_sockets to true in the MachineClass. (Note that the m68k virt machine does not support smp, and therefore does not need that statement.)

Machines also sometimes need to configure associated capabilities in a compatible way. For example, the s390x cpu models may gain new feature flags in newer QEMU releases; when using a compat machine, those new flags need to be off in the cpu models that are used by default.

Inheritance

Compat machines for older machine types need the compatibility changes for newer machine types as well as some changes on top. Typically, this is done by the MachineState respectively MachineClass initializing functions for version n-1 calling the respective initializing functions for version n. As all new compatibility changes are added for the latest versioned machine type, changes are propagated down the whole stack of versions.

All machine types for version n include the hw_compat_<n> array (and the pc_compat_<n> array for x86), unless they are the latest version (which does not need any compat handling yet.) The older compat property arrays are included via the inheritance mechanism.

Putting it all together

QEMU currently supports versioned machine types for x86 (pc-i440fx, pc-q35), arm (virt), aarch64 (virt), s390x (s390-ccw-virtio), ppc64 (pseries), and m68k (virt). At the beginning of each development cycle, new (empty) arrays of compat properties for the last version are added and wired up in the machine types for that last version, new versions of each of these machines are added to the code, and the defaults switched to them (well, that’s the goal.) After that, the framework for adding incompatible changes is in place.

If you find that these changes have not yet been made when you plan to make an incompatible change, it is important that you add the new machine types first.

New and incompatible device properties

If you plan to change the default value of a device property, or add a new property with a default value that will cause guest-observable changes, you need to add an entry that preserves the old value (or sets a value that does not change the behaviour) to the compat property array for the last version. In general (non-x86 specific change), that means adding it to the hw_compat_ array, and all machine types will use it automatically.

Take care to use the right device for specifying the property; for example, there is often some confusion when dealing with virtio devices. If you e.g. modify a virtio-blk property (as in the example above), you need to add a statement for virtio-blk-device and not for virtio-blk-pci, or virtio-blk instances using the ccw or mmio transports would be left out. If, on the other hand, you modify a property only for virtio-blk devices using the pci transport, you need to add a statement for virtio-blk-pci. Similar considerations apply to other devices inheriting from base types.

Per-machine changes

If you change a non-device default characteristic, you need to add a compatibility statement for the machine types for the last version in their instance (or class) init functions. The hardest part here is making sure that all relevant machine types get the update.

For example, if you add a change in the s390x cpu models, it is easy to see that you only need to modify the code for the s390-ccw-virtio machine. For other changes, every versioned machine needs the change. And there are cases like the prefer_sockets change mentioned above, that apply to any machine type that supports smp.

I hope that these explanations help a bit with understanding how machine type compatibility works, and where to add your own changes.

QEMU machine types and compatibility

2022-01-05T12:30:00+01:00

If you want to migrate a guest initially started on an older QEMU version to a newer version of QEMU, you need to make sure that the two machines are actually compatible with each other. Once you exclude things like devices that cannot be migrated at all and make sure both QEMU invocations actually create the same virtual hardware, this basically boils down to using compatible machines.

Versioned machine types

If you simply want to create a machine without any consideration regarding migration compatibility, you will usually do something like

qemu-system-ppc64 -machine pseries (...)

This will create a machine of the pseries type. But in this case, pseries is actually an alias to the latest version of this machine type; for 6.2, this would be pseries-6.2. You can find out which machine types are versioned (and which machine types actually exist for a given binary) via -machine ?:

$ qemu-system-ppc64 -machine ?
Supported machines are:
40p                  IBM RS/6000 7020 (40p)
bamboo               bamboo
g3beige              Heathrow based PowerMAC
mac99                Mac99 based PowerMAC
mpc8544ds            mpc8544ds
none                 empty machine
pegasos2             Genesi/bPlan Pegasos II
powernv10            IBM PowerNV (Non-Virtualized) POWER10
powernv8             IBM PowerNV (Non-Virtualized) POWER8
powernv              IBM PowerNV (Non-Virtualized) POWER9 (alias of powernv9)
powernv9             IBM PowerNV (Non-Virtualized) POWER9
ppce500              generic paravirt e500 platform
pseries-2.1          pSeries Logical Partition (PAPR compliant)
pseries-2.10         pSeries Logical Partition (PAPR compliant)
pseries-2.11         pSeries Logical Partition (PAPR compliant)
pseries-2.12         pSeries Logical Partition (PAPR compliant)
pseries-2.12-sxxm    pSeries Logical Partition (PAPR compliant)
pseries-2.2          pSeries Logical Partition (PAPR compliant)
pseries-2.3          pSeries Logical Partition (PAPR compliant)
pseries-2.4          pSeries Logical Partition (PAPR compliant)
pseries-2.5          pSeries Logical Partition (PAPR compliant)
pseries-2.6          pSeries Logical Partition (PAPR compliant)
pseries-2.7          pSeries Logical Partition (PAPR compliant)
pseries-2.8          pSeries Logical Partition (PAPR compliant)
pseries-2.9          pSeries Logical Partition (PAPR compliant)
pseries-3.0          pSeries Logical Partition (PAPR compliant)
pseries-3.1          pSeries Logical Partition (PAPR compliant)
pseries-4.0          pSeries Logical Partition (PAPR compliant)
pseries-4.1          pSeries Logical Partition (PAPR compliant)
pseries-4.2          pSeries Logical Partition (PAPR compliant)
pseries-5.0          pSeries Logical Partition (PAPR compliant)
pseries-5.1          pSeries Logical Partition (PAPR compliant)
pseries-5.2          pSeries Logical Partition (PAPR compliant)
pseries-6.0          pSeries Logical Partition (PAPR compliant)
pseries-6.1          pSeries Logical Partition (PAPR compliant)
pseries              pSeries Logical Partition (PAPR compliant) (alias of pseries-6.2)
pseries-6.2          pSeries Logical Partition (PAPR compliant) (default)
ref405ep             ref405ep
sam460ex             aCube Sam460ex
taihu                taihu
virtex-ml507         Xilinx Virtex ML507 reference design

As you can see, there are various pseries-x.y machine types for older versions; these are designed to present a configuration that is compatible with a default machine that was created with an older QEMU version. For example, if you wanted to migrate a guest running on a pseries machine that was created using QEMU 5.1, the receiving QEMU would need to be started with

qemu-system-ppc64 -machine pseries-5.1 (...)

Supported machine types

Note: the following applies to upstream QEMU. Distributions may support different versioned machine types in their builds.

This list is as of QEMU 6.2; new versioned machine types may be added in the future, and sometimes old ones deprecated and removed. The machine types for the next QEMU release are usually introduced early in the release cycle (at least, that is the goal…)

arm, aarch64

The virt machine type supports versions since 2.6.

m68k

The virt machine type supports versions since 6.0.

ppc64

The pseries machine type supports versions since 2.1.

s390x

The s390-ccw-virtio machine type supports versions since 2.4.

i386, x86_64

The pc-i440fx machine type supports versions since 1.4 (there used to be even older ones, but they have been removed), while the pc-q35 machine type supports versions since 2.4.

There’s an additional thing to consider here: the pc machine type alias points (as of QEMU 6.2) to the latest pc-i440fx machine type; if you want the latest pc-q35 machine type instead, you have to use q35.

How to use this

If you want to simply fire up a QEMU instance and shut it down again without wanting to migrate it anywhere, you can stick to the default machine type. However, if you might want to migrate the machine later, it is probably a good idea to specify a versioned machine type explicitly, so that you don’t have to remember which QEMU version you started it with.

Or just use management software like libvirt, which will do the machine type expansion to the latest version for you automatically, so you don’t have to worry about it later.

This concludes the usage part of compatible machine types; a follow-up post will look at how this is actually implemented.

Blog update

2021-11-02T00:00:00+01:00

I have moved my blog to a new location and done some other changes at the same time.

This blog is now generated via Jekyll (a huge thank you to the authors here!) This makes posts easier to write for me (especially when formatting command output and similar), and gets rid of intrusive scripts as on the Blogger platform.
This blog’s title is now “KVM, QEMU, and more.” Observant readers may have noticed that I dropped the “Big Iron” part. I may still post s390x-specific content, but in general, I plan to write more about topics that are not architecture-specific.

And yes, I plan to actually post something new this year ;)

s390x changes in QEMU 5.2

2020-11-24T16:34:00+01:00

As, once again, a new QEMU release is around the corner, the time has come to list some s390x changes in there.

TCG has gained emulation support for some additional instructions that had been introduced with the z14. More enhancements needed to be able to run distributions built for the z14 will likely come in the future.
When running under KVM, QEMU now supports the diagnose 0x318 instruction. This can be used to set some diagnostic information (such as the operating system), which may be helpful when servicing the hardware. With this comes support for extended SCCBs; this is needed as the facility indication for diag318 encroaches into the control block used for reporting CPU information. A guest needs support for extended SCCBs to be able to see information for all CPUs if diag318 support is provided.
You can now use virtiofs on s390x, thanks to some endianness fixes, and a vhost-user-fs-ccw device has been added.
Up to now, both fully emulated PCI functions and PCI functions passed via vfio-pci reported the same values when the guest issued CLP instructions. However, the passed through functions may use different values for things such as the supported DMA range. If the host kernel supplies the respective capabilities for the vfio-pci device, QEMU can now provide the real values in the CLP queries.
zPCI is now also able to honour vfio DMA limits, if passed via the vfio-pci device, and can trigger the guest to flush its DMA mappings when needed.
The s390-ccw bios now tries harder to find a bootable device, if the first device is not suitable. This brings s390x booting a bit closer to what other architectures do.
And the usual fixes and cleanups.

Configuring mediated devices (Part 2)

2020-07-29T13:12:00+02:00

In the last part of this article, I talked about configuring a mediated device directly via sysfs. This is a bit cumbersome, and you may want to make your configuration more permanent. Fortunately, there is tooling available for this.

driverctl: bind to the correct driver

driverctl is a tool to manage the driver that a device may bind to. As a device that is supposed to be used via vfio will need to be bound to a vfio driver instead of its ‘normal’ driver, it makes sense to add some configuration that makes sure that this binding is actually done automatically. While driverctl had originally been implemented to work with PCI devices, the css bus (for subchannel devices) supports management with driverctl as of Linux 5.3 as well. (The ap bus for crypto devices does not support setting driver overrides, as it implements a different mechanism.)

Example (vfio-ccw)

Let’s reuse the example from the last post, where we wanted to assign the device behind subchannel 0.0.0313 to the guest. In order to set a driver override, use

[root@host ~]# driverctl -b css set-override 0.0.0313 vfio_ccw

If the subchannel is not currently bound to the vfio-ccw driver already, it will be unbound from its driver and bound to vfio_ccw. Moreover, a udev rule to bind the subchannel to vfio_ccw automatically in the future will be added.

Unfortunately, a word of caution regarding the udev rule is in order: As uevents on the css bus for I/O subchannels are delayed until after device recognition has been performed, automatic binding may not work out as desired. We plan to address that in the future by reworking the way the css bus handles uevents; until then, you may have to trigger a rebind manually. Also, keep in mind that the subchannel id for a device may not be stable (as mentioned previously); automation should be used cautiously in that case.

mdevctl: manage mediated devices

The more tedious part of configuring a passthrough setup is configuring and managing mediated devices. To help with that, mdevctl has been written. It can create, modify, and remove mediated devices (and optionally make those changes persistent), work with configurations and devices created via other means, and list mediated devices and the different types that are supported.

Creating a mediated device

In order to create a mediated device, you need a uuid. You can either provide your own (as in the manual case), or let mdevctl pick one for you. In order to get the same configuration as in the manual configuration examples, let’s create a vfio-ccw device with the same uuid as before.

The following command defines the same mediated device as in the manual example:

[root@host ~]# mdevctl define -u 7e270a25-e163-4922-af60-757fc8ed48c6 -p 0.0.0313 -t vfio_ccw-io -a

Note the ‘-a’, which instructs mdevctl to start the device automatically from now on.

After you’ve created the device, you can check which devices mdevctl is now aware of:

[root@host ~] # mdevctl list -d
7e270a25-e163-4922-af60-757fc8ed48c6 0.0.0313 vfio_ccw-io

Note that the ‘-d’ instructs mdevctl to show defined, but not started devices.

Let’s start the device:

[root@host ~] # mdevctl start -u 7e270a25-e163-4922-af60-757fc8ed48c6
[root@host ~] # mdevctl list -d 7e270a25-e163-4922-af60-757fc8ed48c6 0.0.0313 vfio_ccw-io auto (active)

The mediated device is now ready to be used and can be passed to a guest.

Making your configuration persistent

If you already created a mediated device manually, you may want to reuse the existing configuration and make it persistent, instead of starting from scratch.

So, let’s create another vfio-ccw the manual way:

[root@host ~] # uuidgen
b29e4ca9-5cdb-4ee1-a01b-79085b9ab237
[root@host ~] # echo "b29e4ca9-5cdb-4ee1-a01b-79085b9ab237" > /sys/bus/css/drivers/vfio_ccw/0.0.0314/mdev_supported_types/vfio_ccw-io/create

mdevctl now actually knows about the active device (in addition to the device we configured before):

[root@host ~] # mdevctl list
b29e4ca9-5cdb-4ee1-a01b-79085b9ab237 0.0.0314 vfio_ccw-io
7e270a25-e163-4922-af60-757fc8ed48c6 0.0.0313 vfio_ccw-io (defined)

But it obviously does not have a definition for the manually created device:

[root@host ~] # mdevctl list -d
7e270a25-e163-4922-af60-757fc8ed48c6 0.0.0313 vfio_ccw-io auto (active)

On a restart, the new device would be gone again; but we can make it persistent:

[root@host ~] # mdevctl define -u b29e4ca9-5cdb-4ee1-a01b-79085b9ab237
[root@host ~ ] mdevctl list
b29e4ca9-5cdb-4ee1-a01b-79085b9ab237 0.0.0314 vfio_ccw-io (defined)
7e270a25-e163-4922-af60-757fc8ed48c6 0.0.0313 vfio_ccw-io (defined)

If you check under /etc/mdevctl.d/, you will find that an appropriate JSON file has been created:

[root@host ~] # cat /etc/mdevctl.d/0.0.0314/b29e4ca9-5cdb-4ee1-a01b-79085b9ab237
{
	"mdev_type": "vfio_ccw-io",
	"start": "manual",
	"attrs": []
}

(Note that this device is not automatically started by default.)

Modifying an existing device

There are good reasons to modify an existing device: you may want to modify your setup, or, in the case of vfio-ap, you need to modify some attributes before being able to use the device in the first place.

Let’s first create the device. This command creates the same device as created manually in the last post:

[root@host ~] # mdevctl define -u "669d9b23-fe1b-4ecb-be08-a2fabca99b71" --parent matrix --type vfio_ap-passthrough
[root@host ~] # mdevctl list -d
669d9b23-fe1b-4ecb-be08-a2fabca99b71 matrix vfio_ap-passthrough manual

This device is not yet very useful, as you still need to assign some queues to it. It now looks like this:

[root@host ~] # mdevctl list -d -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71 --dumpjson
{
	"mdev_type": "vfio_ap-passthrough",
	"start": "manual"
}

Let’s modify the device and add some queues:

[root@host ~] #&nbsp;mdevctl modify -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71 --addattr=assign_adapter --value=5
[root@host ~] #&nbsp;mdevctl modify -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71 --addattr=assign_domain --value=4
[root@host ~] #&nbsp;mdevctl modify -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71 --addattr=assign_domain --value=0xab

The device’s JSON now looks like this:

[root@host ~] # mdevctl list -d -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71 --dumpjson
{
	"mdev_type": "vfio_ap-passthrough",
	"start": "manual",
	"attrs": [
		{
			"assign_adapter": "5"
		},
		{
			"assign_domain": "4"
		},
		{
			"assign_domain": "0xab"
		}
	]
}

This is now exactly what we had defined manually in the last post.

But what if you notice that you want domain 0x42 instead of domain 4? Just modify the definition. To make it easier to figure out how to specify the attribute to manipulate, use this output:

[root@host ~] # devctl list -dv -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71
669d9b23-fe1b-4ecb-be08-a2fabca99b71 matrix vfio_ap-passthrough manual
Attrs:
	@{0}: {"assign_adapter":"5"}
	@{1}: {"assign_domain":"4"}
	@{2}: {"assign_domain":"0xab"}

You want to remove attribute 1, and add a new value:

[root@host ~] # mdevctl modify -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71 --delattr --index=1
[root@host ~] # mdevctl modify -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71 --addattr=assign_domain --value=0x42

Let’s check that it now looks as desired:

[root@host ~] # mdevctl list -dv -u 669d9b23-fe1b-4ecb-be08-a2fabca99b71
669d9b23-fe1b-4ecb-be08-a2fabca99b71 matrix vfio_ap-passthrough manual
Attrs:
	@{0}: {"assign_adapter":"5"}
	@{1}: {"assign_domain":"0xab"}
	@{2}: {"assign_domain":"0x42"}

Future development

While mdevctl works perfectly fine for managing individual mediated devices, it does not maintain a view of the complete system. This means you notice conflicts between two devices only when you try to activate the second one. In the case of vfio-ap, the rules to be considered are complex, and there is quite some potential for conflict. In order to be able to catch that kind of problem early, we plan to add callouts to mdevctl, which would e.g. allow to invoke a tool for validation when a new device is added, but before it is activated. This is potentially useful for other device types as well.

Configuring mediated devices (Part 1)

2020-07-27T18:55:00+02:00

vfio-mdev has become popular over the last few years for assigning certain classes of devices to guests. On the s390x side, vfio-ccw and vfio-ap are using the vfio-mdev framework for making channel devices and crypto adapters accessible to guests.

This and a follow-up article aim to give an overview of the infrastructure, how to set up and manage devices, and how to use tooling for this.

What is a mediated device?

A general overview

Mediated devices grew out of the need to build upon the existing vfio infrastructure in order to support more fine grained management of resources. Some of the initial use cases included GPUs and (maybe somewhat surprisingly) s390 channel devices.

When using the mediated device (mdev) API, common tasks are performed in the mdev core driver (like device management), while device-specific tasks are done in a vendor driver. Current in-kernel examples of vendor drivers are the Intel vGPU driver, vfio-ccw, and vfio-ap.

Examples on s390

vfio-ccw

vfio-ccw can be used to assign channel devices. It is pretty straightforward: vfio-ccw is an alternative driver for I/O subchannels, and a single mediated device per subchannel is supported.

vfio-ap

vfio-ap can be used to assign crypto cards/queues (APQNs). It is a bit more involved, requiring prior setup on the ap bus level and configuration of a ‘matrix’ device. Complex relationships between the resources that can be assigned to different guests exist. Configuration-wise, this is probably the most complex mediated device available today.

Configuring a mediated device: the manual way

Mediated devices can be configured manually via sysfs operations. This is a good way to see what actually happens, but probably not what you want to do as a general administration task. Tools to help here will be introduced in part 2 of this article.

I will show the steps for both vfio-ccw and vfio-ap, just to show two different approaches. (Both examples are also used in the QEMU documentation, in case this looks familiar.)

Binding to the correct driver

vfio-ccw

Assume you want to use a DASD with the device bus ID 0.0.2b09. As vfio-ccw operates on the subchannel level, you first need to locate the subchannel for this device:

[root@host ~]# lscss | grep 0.0.2b09 | awk '{print $2}'
0.0.0313

(A word of caution: a device is not guaranteed to use the same subchannel at all times; on LPARs, the subchannel number will usually be stable, but z/VM – and QEMU – assign subchannel numbers in a consecutive order. If you don’t get any hotplug events for a device, the subchannel number will stay stable for at least as long as the guest is running, though.)

Now you need to unbind the subchannel device from the default I/O subchannel driver and bind it to the vfio-ccw driver (make sure the device is not in use!):

[root@host ~]# echo 0.0.0313 > /sys/bus/css/devices/0.0.0313/driver/unbind
[root@host ~]# echo 0.0.0313 > /sys/bus/css/drivers/vfio_ccw/bind

vfio-ap

You need to perform some preliminary configuration of your crypto adapters before you can use any of them with vfio-ap. If nothing different has been set up, a crypto adapter will only bind to the default device drivers, and you cannot use it via vfio-ap. In order to be able to bind an adapter to vfio-ap, you first need to modify the /sys/bus/ap/apmask and /sys/bus/ap/aqmask entries. Both are basically bitmasks that indicate that the matching adapter IDs respectively queue indices can only be bound to the default drivers. If you want to use a certain APQN via vfio-ap, you need to unset the respective bits.

Let’s assume you want to assign the APQNs (5, 4) and (5, ab). First, you need to make the adapter and the domains available to non-default drivers:

[root@host ~]# echo -5 > /sys/bus/ap/apmask
[root@host ~]# echo -4, -0xab > /sys/bus/ap/aqmask

This should result in the devices being bound to the vfio_ap driver (you can verify this by looking for them under /sys/bus/ap/drivers/vfio_ap/).

Create a mediated device

The basic workflow is “pick a uuid, create a mediated device identified by it”.

vfio-ccw

For vfio-ccw, the two steps of the basic workflow are enough:

[root@host ~]# uuidgen
7e270a25-e163-4922-af60-757fc8ed48c6
[root@host ~]# echo "7e270a25-e163-4922-af60-757fc8ed48c6" > /sys/bus/css/devices/0.0.0313/mdev_supported_types/vfio_ccw-io/create

vfio-ap

For vfio-ap, you need a more involved approach. The uuid is used to create a mediated device under the ‘matrix’ device:

[root@host ~] # uuidgen
669d9b23-fe1b-4ecb-be08-a2fabca99b71
[root@host ~]# echo "669d9b23-fe1b-4ecb-be08-a2fabca99b71" > /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create

This mediated device will need to collect all APQNs that you want to pass to a specific guest. For that, you need to use the assign_adapter, assign_domain, and possibly assign_control_domain attributes (we’ll ignore control domains for simplicity’s sake.) All attributes have a companion unassign_ attribute to remove adapters/domains from the mediated device again. You can only assign adapters/domains that you removed from apmask/aqmask in the previous step. To follow up on our example again:

[root@host ~]# echo 5 > /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/669d9b23-fe1b-4ecb-be08-a2fabca99b71/assign_adapter
[root@host ~]# echo 4 > /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/669d9b23-fe1b-4ecb-be08-a2fabca99b71/assign_domain
[root@host ~]# echo 0xab > /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/669d9b23-fe1b-4ecb-be08-a2fabca99b71/assign_domain

If you want to make sure that the mediated device is set up correctly, check via

[root@host ~]# cat /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/669d9b23-fe1b-4ecb-be08-a2fabca99b71/matrix
05.0004
05.00ab

Configuring QEMU/libvirt

Your mediated device is now ready to be passed to a guest.

vfio-ccw

Let’s assume you want the device to show up as device 0.0.1234 in the guest.

For the QEMU command line, use

-device vfio-ccw,devno=fe.0.1234,sysfsdev=/sys/bus/mdev/devices/7e270a25-e163-4922-af60-757fc8ed48c6

For libvirt, use the following XML snippet in the <devices> section:

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ccw'>
	<source>
		<address uuid='7e270a25-e163-4922-af60-757fc8ed48c6'/>
	</source>
	<address type='ccw' cssid='0xfe' ssid='0x0' devno='0x1234'/>
</hostdev>

vfio-ap

Any APQNs will show up in the guest exactly as they show up in the host (i.e., no remapping is possible.)

For the QEMU command line, use

-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/669d9b23-fe1b-4ecb-be08-a2fabca99b71

For libvirt, use the following XML snippet in the <devices> section:

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
	<source>
		<address uuid='669d9b23-fe1b-4ecb-be08-a2fabca99b71'/>
	</source>
</hostdev>

Tooling

All this manual setup is a bit tedious; the next article in this series will look at some of the tooling that is available for mediated devices.

s390x changes in QEMU 5.1

2020-07-10T16:27:00+02:00

QEMU has entered softfreeze for 5.1, so it is time to summarize the s390x changes in that version.

Protected virtualization

One of the biggest features on the s390/KVM side in Linux 5.7 had been protected virtualization aka secure execution, which basically restricts the (untrusted) hypervisor from accessing all of the guest’s memory and delegates many tasks to the (trusted) ultravisor. QEMU 5.1 introduces the QEMU part of the feature.

In order to be able to run protected guests, you need to run on a z15 or a Linux One III, with at least a 5.7 kernel. You also need an up-to-date s390-tools installation. Some details are available in the QEMU documentation. For more information about what protected virtualization is, watch this talk from KVM Forum 2019 and this talk from 36C3.

vfio-ccw

vfio-ccw has also seen some improvements over the last release cycle.

Requests that do not explicitly allow prefetching in the ORB are no longer rejected out of hand (although the kernel may still do so, if you run a pre-5.7 version.) The rationale behind this is that most device drivers never modify their channel programs dynamically, and the one common code path that does (IPL from DASD) is already accommodated by the s390-ccw bios. While you can instruct QEMU to ignore the prefetch requirement for selected devices, this is an additional administrative complication for little benefit; it is therefore no longer required.
In order to be able to relay changes in channel path status to the guest, two new regions have been added: a schib region to relay real data to stsch, and a crw region to relay channel reports. If, for example, a channel path is varied off on the host, all guests using a vfio-ccw device that uses this channel path now get a proper channel report for it.

Other changes

Other than the bigger features mentioned above, there have been the usual fixes, improvements, and cleanups, both in the main s390x QEMU code and in the s390-ccw bios.

s390x changes in QEMU 5.0

2020-04-08T13:58:00+02:00

QEMU is currently in hardfreeze, with the 5.0 release expected at the end of the month. Here’s a quick list of some notable s390x changes.

You can finally enable Adapter Interrupt Suppression in the cpu model (ais=on) when running under KVM. This had been working under TCG for some time now, but KVM was missing an interface that was provided later – and we finally actually check for that interface in QEMU. This is mostly interesting for PCI.
QEMU had been silently fixing odd memory sizes to something that can be reported via SCLP for some time. Silently changing user input is probably not such a good idea; compat machines will continue to do so to enable migration from old QEMUs for machines with odd sizes, but will print a warning now. If you have such an old machine (and you can modify it), it might be a good idea to either specify the memory size it gets rounded to or to switch to the 5.0 machine type, where memory sizes can be more finegrained due to the removal of support for memory hotplug. We may want to get rid of the code doing the fixup at some time in the future.
QEMU now properly performs the whole set of initial, clear, and normal cpu reset.
And the usual fixes, cleanups, and improvements.

For 5.1, expect more changes; support for protected virtualization will be a big item.

Channel Measurements: A Quick Overview

2020-01-22T14:42:00+01:00

The s390 channel subsystem can gather some statistics on I/O performance for you, which might be useful if you try to figure out why something is not performing as well as you’d expect it to be. From a QEMU/KVM perspective, this is currently mainly useful on the host.

Channel monitoring for ccw devices

The first kind of channel measurements is those collected per subchannel. For a detailed overview of what actually happens there, turn to the Principles of Operation, Chapter 17 (“I/O Support Functions”), “Channel Monitoring”. I’ll cover here what will most likely be of interest to people running a Linux (host) system.

Enabling channel measurements

If you a running a non-vintage machine (i.e. a z990 or later), you will not need a system-wide setup. Older machines should be fine as well, if you do not want to measure more than 1024 devices.

To enable measurements for a specific ccw device (say, 0.0.1234), simply issue:

chccwdev -a cmb_enable=1 0.0.1234

Measurements collected

Under /sys/bus/ccw/device/0.0.1234/, you should now have a new subdirectory called cmf, which contains some files. For a system that has been running for some time, the contents may look something like the following:

head cmf/*
==> cmf/avg_control_unit_queuing_time <==
0
==> cmf/avg_device_active_only_time <==
0
==> cmf/avg_device_busy_time <==
0
==> cmf/avg_device_connect_time <==
829031
==> cmf/avg_device_disconnect_time <==
398526
==> cmf/avg_function_pending_time <==
142810
==> cmf/avg_initial_command_response_time <==
19170
==> cmf/avg_sample_interval <==
8401681344
==> cmf/avg_utilization <==
00.0%
==> cmf/sample_count <==
10803
==> cmf/ssch_rsch_count <==
10803

Note that all values but sample_count and ssch_rsch_count are averaged over time. We also see that samples seem to have been taken whenever the driver issued a ssch.

The device in our example shows an avg_utilization of 0%, which is consistent with a device that mostly sits idle. But what about a device where something is actually happening?

head cmf/*
==> cmf/avg_control_unit_queuing_time <==
0
==> cmf/avg_device_active_only_time <==
0
==> cmf/avg_device_busy_time <==
0
==> cmf/avg_device_connect_time <==
58454
==> cmf/avg_device_disconnect_time <==
16743818
==> cmf/avg_function_pending_time <==
99322
==> cmf/avg_initial_command_response_time <==
20284
==> cmf/avg_sample_interval <==
153014636
==> cmf/avg_utilization <==
11.0%
==> cmf/sample_count <==
1281
==> cmf/ssch_rsch_count <==
1281

Here, we see a higher avg_utilization, but actually not that many ssch invocations. Interesting is the relatively high value of avg_device_disconnect_time: It indicates that there are quite long intervals where the device and the channel subsystem do not talk to each other. That might, for example, happen if other LPARs on the same system drive a lot of I/O via the same channel paths as the device.

Help, I cannot enable channel measurements on my device!

There’s one drawback when trying to enable channel measurements on a live device: It needs to execute a msch, which only can be done on an idle subchannel. For devices that execute separate ssch invocations to go about their business (e.g. dasd), the common I/O layer can squeeze in the msch between ssch invocations and all is well. However, some devices use a long-running channel program, which will not conclude during the time the device is enabled; the most prominent example are devices using QDIO, like zFCP adapters or OSA cards. In that case, the common I/O layer cannot squeeze in a msch; you might try disabling the device, but that’s usually not something you want to do in a live system.

Extended channel measurements

What if you want to find out something not about an individual device, but for a channel path? There’s a feature for that; you can issue

echo 1 > /sys/devices/css0/cm_enable

and will find new entries (measurement, measurement_chars) under the various chp0.xx objects.

Unfortunately, these attributes only provide some binary data, which does not seem to be publicly documented, and I’m not aware of any tool that can parse them.

Channel measurements in QEMU guests

So far, all measurements have been collected on the host; but what about measurements in the guest?

The good news: You can turn on channel measurements for ccw devices in the guest. The bad news: They are not very useful.

Consider, for example, this virtio-ccw device:

head cmf/*
==> cmf/avg_control_unit_queuing_time <==
0
==> cmf/avg_device_active_only_time <==
0
==> cmf/avg_device_busy_time <==
0
==> cmf/avg_device_connect_time <==
0
==> cmf/avg_device_disconnect_time <==
0
==> cmf/avg_function_pending_time <==
0
==> cmf/avg_initial_command_response_time <==
0
==> cmf/avg_sample_interval <==
-1
==> cmf/avg_utilization <==
00.0%
==> cmf/sample_count <==
0
==> cmf/ssch_rsch_count <==
134

No samples, just a ssch count. Why? QEMU does not fully emulate the sampling infrastructure; only counting of ssch is done (which is very easy to implement). Moreover, virtio-ccw devices use channel programs mainly to set up queues, negotiate features, etc., so measurements here do not reflect what is going on on the virtqueues, which would be the interesting part for performance issues.

But what about a dasd passed through via vfio-ccw? That one should have more statistics, right?

head cmf/*
==> cmf/avg_control_unit_queuing_time <==
0
==> cmf/avg_device_active_only_time <==
0
==> cmf/avg_device_busy_time <==
0
==> cmf/avg_device_connect_time <==
0
==> cmf/avg_device_disconnect_time <==
0
==> cmf/avg_function_pending_time <==
0
==> cmf/avg_initial_command_response_time <==
0
==> cmf/avg_sample_interval <==
-1
==> cmf/avg_utilization <==
00.0%
==> cmf/sample_count <==
0
==> cmf/ssch_rsch_count <==
144

No samples, just a ssch count, again. Why? Currently, vfio-ccw uses the same emulation infrastructure as the other emulated devices. In the future, we may implement some kind of passthrough for channel measurements, but that requires some work.