A couple of times I already ran into the situation that I wanted to provide
a small guest disk image to other people. For example, one time I wanted to
provide a test application like LTP to colleagues via a server where I only
had some limited disk quota available. Back then I was still able to resolve
the problem by installing a stock Linux distribution together with the test
software into a normal qcow2 image, and then to shrink the image with
qemu-img convert and
xz to approximately 500 MiB.
But when I started to think about the QEMU advent calendar 2018, where I wanted to provide many small images for various different target architectures, it was clear to me that I needed a different approach. First the disk images needed to be much smaller due to network traffic constraints, and for many of “non-mainstream” target architectures (like MicroBlaze or Xtensa) you also can not easily get a standard Linux distribution that installs without problems on the machines that QEMU provides.
Instead of using a pre-built Linux distribution, it would also be possible to cross-compile the kernel and user space programs and build a small disk image with that on your own. However, figuring out how to do that for multiple target architectures would have been very cumbersome and time consuming.
So after doing some research, I finally discovered buildroot, which is an excellent framework for doing exactly what I wanted: It allows to create small disk images for non-x86 target CPUs, with all the magic about cross compiling and image creation wrapped into its internal scripts, and with a very flexible Kconfig-style configuration system on top.
For those who are interested, here’s now a short description how to use buildroot for creating a small guest disk image:
- Download the version that you like to use from the
buildroot download page and unpack it:
$ wget https://buildroot.org/downloads/buildroot-2018.02.9.tar.bz2 $ tar -xaf buildroot-2018.02.9.tar.bz2 $ cd buildroot-2018.02.9/
- Now you have to choose for which CPU and machine target you want to
build. Have a look at the pre-defined config files and then select one.
In the following example, I’m going to use the “pseries” POWER machine:
$ cd configs/ $ ls qemu* qemu_aarch64_virt_defconfig qemu_nios2_10m50_defconfig qemu_arm_versatile_defconfig qemu_or1k_defconfig qemu_arm_versatile_nommu_defconfig qemu_ppc64le_pseries_defconfig qemu_arm_vexpress_defconfig qemu_ppc64_pseries_defconfig qemu_m68k_mcf5208_defconfig qemu_ppc_g3beige_defconfig qemu_m68k_q800_defconfig qemu_ppc_mpc8544ds_defconfig qemu_microblazebe_mmu_defconfig qemu_ppc_virtex_ml507_defconfig qemu_microblazeel_mmu_defconfig qemu_sh4eb_r2d_defconfig qemu_mips32r2el_malta_defconfig qemu_sh4_r2d_defconfig qemu_mips32r2_malta_defconfig qemu_sparc64_sun4u_defconfig qemu_mips32r6el_malta_defconfig qemu_sparc_ss10_defconfig qemu_mips32r6_malta_defconfig qemu_x86_64_defconfig qemu_mips64el_malta_defconfig qemu_x86_defconfig qemu_mips64_malta_defconfig qemu_xtensa_lx60_defconfig qemu_mips64r6el_malta_defconfig qemu_xtensa_lx60_nommu_defconfig qemu_mips64r6_malta_defconfig $ cd .. $ make qemu_ppc64_pseries_defconfig
- Now run
make menuconfigto fine tune your build. I recommend to have a look at the following settings first:
- In the Toolchain section, you might need to enable other languages like C++ in case it is required for the application that you want to ship in the image.
- In the System Configuration section, change the System Banner to something that better suits your disk image.
- Check the Kernel section to see whether the right kernel settings are used here. The defaults should be fine most of the time, but in case you want to use a newer kernel version for example, or a different kernel config file, you can adjust it here. Note that you also should adjust the kernel header version in the Toolchain section if you change the kernel version here.
- Have a look at the Target packages section – maybe the application that you want to include is already available by the base buildroot system. In that case you can already enable it here.
- Check the Filesystem images section and decide which kind of image you want to ship later. For example, for most of the QEMU advent calendar images, I used a simple initrd only, so I unchecked the ext2/3/4 root filesystem here and used initial RAM filesystem linked into linux kernel instead.
Now save your configuration, exit the config menu, and type
makefor a first test to see whether it produces a usable image. Note: Don’t use the
-jparameter of make here, buildroot will figure that out on its own instead.
- Once the build finished successfully, have a look at the
output/images/directory. You can start your guest with the results from there to give it a try. For example if you built with the ppc64 pseries configuration, with the initrd linked into the kernel:
$ qemu-system-ppc64 -M pseries -m 1G -kernel output/images/vmlinux
You should see the kernel booting up, and if you have a look at the serial console, there is also a getty running where you can log in as root and look around.
- To customize your build, you sooner or later want to add additional files
to the image, for example some additional init scripts in the /etc/init.d/
folder. Or in the above case, it would be good to also have getty running
on the graphical console. So to add custom files, the best way is to create
an overlay folder which will be copied into the destination filesystem during
the make process:
$ mkdir overlay/etc/init.d $ cp my-startup-script.sh overlay/etc/init.d/S99myscript # If you have one $ cp output/target/etc/inittab overlay/etc/inittab $ echo 'tty1::respawn:/sbin/getty -L tty1 0 linux' >> overlay/etc/inittab
make menuconfigand set the Root filesystem overlay directories option in the System Configuration section to the
overlayfolder that you have just created. Run
makeagain and the next time you start your guest, you should see the new files in the image, e.g. also a getty running on the graphical console. Note: Do not try to add/change files directly in the output/target/ folder. That looks tempting first, but this is just a temporary folder used by the build system, which can be overwritten at any time and will be erased when you run
make cleanfor example.
If you need to tweak the kernel configuration, you can run
make linux-menuconfigand do the appropriate changes there. For example, if you want to get keyboard input for the ppc64 pseries machine on the graphical console, you should enable the USB XHCI driver in the kernel, too. Once you are happy with the kernel configuration, save it, exit the menu and type
make linux-rebuild && make. Note: To avoid that the kernel config gets reset after you run
make cleanat a later point in time, you should copy output/build/linux-*/.config to a safe location. Then run
make menuconfig, change the Kernel -> Kernel configuration setting to Use a custom config file and set the Configuration file path to the copied file.
- If you want to add additional software to your image, you basically have to provide a Config.in file and a *.mk file. I recommend to have a look at the various packages in the package/ directory. Use one of the software from there with a similar build system as a template, and have a closer look at buildroot manual for details. Tweaking the build system of your software to properly cross-compile can be a little bit tricky some times, but most software that uses standard systems like autoconf should be fine.
That’s it. You now should be able to package your software in really small VM images. Of course, there are still lots of other settings that you can tweak in the buildroot environment – if you need any of these just have a look at the good buildroot manual for more information.