How to use secure RHCOS images on s390x
Recently, I needed to debug a problem that only occurred in RHCOS images that are running in secure execution mode on an IBM Z system. Since I don’t have a OCP installation at hand, I wanted to run such an image directly with QEMU or libvirt. This sounded easy at a first glance, since there are qcow2 images available for RHCOS, but in the end, it was quite tricky to get this working, so I’d like to summarize the steps here, maybe it’s helpful for somebody else, too. Since the “secex” images are encrypted, you cannot play the usual tricks with e.g. guestfs here, you have to go through the ignition process of the image first. Well, maybe there is already the right documentation for this available somewhere and I missed it, but most other documents mainly talk about x86 or normal (unencrypted) images (like the one for Fedora CoreOS on libvirt ), so I think it will be helpful to have this summary here anyway.
Preparation
First, make sure that you have the right tools installed for this task:
Since we are interested in the secure execution image, we have to download the image with “secex” in the name, together with the right GPG key that is required for encrypting the config file later, for example:
wget https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/4.16/4.16.3/rhcos-qemu-secex.s390x.qcow2.gz
wget https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/4.16/4.16.3/rhcos-ignition-secex-key.gpg.pub
Finally, uncompress the image. And since we want to avoid modifying the original image, let’s also create an overlay qcow2 image for it:
Creation of the configuration file
For being able to log in your guest via ssh later, you need an ssh key, so let’s create one and add it to your local ssh-agent:
If you also want to log in on the console via password, create a
password hash with the mkpasswd
tool, too.
Now create a butane configuration file and save it as “config.bu
”:
variant: fcos
version: 1.4.0
passwd:
users:
- name: core
ssh_authorized_keys:
- INSERT_THE_CONTENTS_OF_YOUR_rhcos-key.pub_FILE_HERE
password_hash: INSERT_THE_HASH_FROM_mkpasswd_HERE
groups:
- wheel
storage:
files:
- path: /etc/se-hostkeys/ibm-z-hostkey-1
overwrite: true
contents:
local: HKD.crt
systemd:
units:
- name: serial-getty@.service
mask: false
Make sure to replace the “INSERT_…” markers in the file with the contents
of your rhcos-key.pub
and the hash from mkpasswd
, and also
make sure to have the host key document (required for encrypting the
guest with genprotimg
) available as HKD.crt
in the current directory.
Next, the butane config file needs to be converted into an ignition file, which then needs to be encrypted with the GPG key of the RHCOS image:
Ignition of the guest image
The encrypted config file can now be used to start the ignition of the guest. On s390x, the config file is not presented via the “fw_cfg” mechanism to the guest (like it is done on x86), but with a drive that has a special serial number. Thus QEMU should be started like this:
This should start the ignition process during the first boot of the guest.
During future boots of the guest, you don’t have to specify the drive with
the “config.crypted” file anymore.
Once the ignition is done, you can log in to the guest either on the
console with the password that you created with mkpasswd
, or via ssh:
Now you should be able to use the image. But keep in mind that this is an
rpm-ostree based image, so for installing additional packages, you have to
use rpm-ostree install
instead of dnf install
here. And the kernel
can be replaced like this, for example:
That’s it! Now you can enjoy your configured secure-execution RHCOS image!
Special thanks to Nikita D. for helping me understanding the ignition process of the secure execution images.