Channel I/O: Talking to devices

Having a nice set of channel devices available to your OS is all fine and good; but how do you actually talk to them? This post attempts to give a high-level overview, while also explaining some more acronyms.

Let’s look again at the example configuration from the last post:

Device   Subchan.  DevType CU Type Use  PIM PAM POM  CHPIDs
----------------------------------------------------------------------
0.0.0000 0.0.0000  0000/00 3832/01 yes  80  80  ff   00000000 00000000
0.0.0042 0.0.0001  0000/00 3832/02 yes  80  80  ff   00000000 00000000

The second device is the virtio-blk device 0.0.0042 on subchannel 0.0.0001, having channel path 0. Being virtio, this is a very simplified variation of what you’d see on real hardware (although this also can be a benefit in some way). Think of it as the following:

Device 0.0.0042 is accessed via channel path 0, and subchannel 0.0.0001 is used as a means to address it.

The access (channel path) is configured in the hypervisor (or in the hardware definitions). The subchannel is what the OS will use as a target for I/O instructions and how it can associate I/O interrupts with the device they are for.

I/O instructions? There’s a whole zoo of them, but they share some characteristics:

• They take a subchannel identifier as parameter.
• They are privileged: I.e., on a Linux system, they can only be issued by the kernel and not from user space.

Check the Principles of Operation (SA22-7832), chapter 14, for the whole story. Here, I’ll concentrate just on two instructions:

• START SUBCHANNEL (SSCH) - start a channel program
• TEST SUBCHANNEL (TSCH) - retrieve subchannel status

So, what’s a channel program? It’s basically a set of instructions sent to the control unit and executed. You can even branch in them. The basic unit of those instructions is the channel command word (CCW). The ccw is such a basic characteristic of channel I/O that it is used throughout Linux and QEMU for channel devices: For example, Linux has ccw_devices on the ccw bus, and QEMU has CcwDevices, most notably VirtIOCcwDevices (like the devices in the example).

ccws consist of three parts:

• The command. This falls into the categories of read (read data from the device), write (write data to the device) or control (for example, rewinding a tape). An 8-bit value.
• The flags, which control error handling or program flow. I’ll ignore them for simplicity here.
• The data address. This is an address in memory where data is written to (read) or read from (write).

Let’s take an example. The SENSE ID command is a basic operation supported by both virtual devices like the virtio devices and real hardware devices. It is used to obtain configuration information from the device, like the CU type information in the output above. It is usually the first ccw an operating system issues to the device.

The operating system will assemble a ccw: The command code will be 0xe4 for SENSE ID, and the data address will point to a location wherethe OS wants to have the obtained information. The OS will also assemble a so-called ORB (operation request block), which, amongst other things, points to the assembled ccw (respectively the first one in a chain). This ORB and the subchannel id are the two parameters for the SSCH instruction. If all goes well, the OS will receive a condition code 0 and knows that it will be signalled asynchronously once the channel program has been processed (successfully or with errors)¹.

Processing of the actual channel command is done asynchronously by real hardware (QEMU does it synchronously for simplicity reasons). The result is that the wanted data is put into the memory area refered to by the ccw². Subsequently, the subchannel is made status pending: Information is ready for retrieval by the OS.

Usually, the OS wants to have a notification that the subchannel became status pending; this is done via an I/O interrupt. I/O interrupts on s390 carry extra status which is written to the low memory area of the cpu receiving the interrupt; amongst other things, this status contains the subchannel id.

Next, the OS needs to actually retreive the status information: This is done via the TSCH instruction, which in turn makes the subchannel no longer status pending and ready for the next I/O request via SSCH. The status contains enough information for the OS to determine whether the request was successful (and the sense id information has been stored), or whether there was an error.³

Of course, this is all only scratching at the surface of channel programs; interested readers can peek at the Linux kernel and QEMU to get a feel for both parts or at the Principles of Operation for the whole story.⁴