A couple of times I already ran into the situation that I wanted to provide a small guest disk image to other people. For example, one time I wanted to provide a test application like LTP to colleagues via a server where I only had some limited disk quota available. Back then I was still able to resolve the problem by installing a stock Linux distribution together with the test software into a normal qcow2 image, and then to shrink the image with qemu-img convert and xz to approximately 500 MiB.

But when I started to think about the QEMU advent calendar 2018, where I wanted to provide many small images for various different target architectures, it was clear to me that I needed a different approach. First the disk images needed to be much smaller due to network traffic constraints, and for many of “non-mainstream” target architectures (like MicroBlaze or Xtensa) you also can not easily get a standard Linux distribution that installs without problems on the machines that QEMU provides.

Instead of using a pre-built Linux distribution, it would also be possible to cross-compile the kernel and user space programs and build a small disk image with that on your own. However, figuring out how to do that for multiple target architectures would have been very cumbersome and time consuming.

So after doing some research, I finally discovered buildroot, which is an excellent framework for doing exactly what I wanted: It allows to create small disk images for non-x86 target CPUs, with all the magic about cross compiling and image creation wrapped into its internal scripts, and with a very flexible Kconfig-style configuration system on top.

For those who are interested, here’s now a short description how to use buildroot for creating a small guest disk image:

  1. Download the version that you like to use from the buildroot download page and unpack it: 
    $ wget https://buildroot.org/downloads/buildroot-2018.02.9.tar.bz2
$ tar -xaf buildroot-2018.02.9.tar.bz2 
$ cd buildroot-2018.02.9/
  2. Now you have to choose for which CPU and machine target you want to build. Have a look at the pre-defined config files and then select one. In the following example, I’m going to use the “pseries” POWER machine: 
    $ cd configs/
$ ls qemu*
qemu_aarch64_virt_defconfig         qemu_nios2_10m50_defconfig
qemu_arm_versatile_defconfig        qemu_or1k_defconfig
qemu_arm_versatile_nommu_defconfig  qemu_ppc64le_pseries_defconfig
qemu_arm_vexpress_defconfig         qemu_ppc64_pseries_defconfig
qemu_m68k_mcf5208_defconfig         qemu_ppc_g3beige_defconfig
qemu_m68k_q800_defconfig            qemu_ppc_mpc8544ds_defconfig
qemu_microblazebe_mmu_defconfig     qemu_ppc_virtex_ml507_defconfig
qemu_microblazeel_mmu_defconfig     qemu_sh4eb_r2d_defconfig
qemu_mips32r2el_malta_defconfig     qemu_sh4_r2d_defconfig
qemu_mips32r2_malta_defconfig       qemu_sparc64_sun4u_defconfig
qemu_mips32r6el_malta_defconfig     qemu_sparc_ss10_defconfig
qemu_mips32r6_malta_defconfig       qemu_x86_64_defconfig
qemu_mips64el_malta_defconfig       qemu_x86_defconfig
qemu_mips64_malta_defconfig         qemu_xtensa_lx60_defconfig
qemu_mips64r6el_malta_defconfig     qemu_xtensa_lx60_nommu_defconfig
qemu_mips64r6_malta_defconfig
$ cd ..
$ make qemu_ppc64_pseries_defconfig
  3. Now run make menuconfig to fine tune your build. I recommend to have a look at the following settings first:
    • In the Toolchain section, you might need to enable other languages like C++ in case it is required for the application that you want to ship in the image.
    • In the System Configuration section, change the System Banner to something that better suits your disk image.
    • Check the Kernel section to see whether the right kernel settings are used here. The defaults should be fine most of the time, but in case you want to use a newer kernel version for example, or a different kernel config file, you can adjust it here. Note that you also should adjust the kernel header version in the Toolchain section if you change the kernel version here.
    • Have a look at the Target packages section – maybe the application that you want to include is already available by the base buildroot system. In that case you can already enable it here.
    • Check the Filesystem images section and decide which kind of image you want to ship later. For example, for most of the QEMU advent calendar images, I used a simple initrd only, so I unchecked the ext2/3/4 root filesystem here and used initial RAM filesystem linked into linux kernel instead.

  4. Now save your configuration, exit the config menu, and type make for a first test to see whether it produces a usable image. Note: Don’t use the -j parameter of make here, buildroot will figure that out on its own instead.

  5. Once the build finished successfully, have a look at the output/images/ directory. You can start your guest with the results from there to give it a try. For example if you built with the ppc64 pseries configuration, with the initrd linked into the kernel: 
    $ qemu-system-ppc64 -M pseries -m 1G -kernel output/images/vmlinux

    You should see the kernel booting up, and if you have a look at the serial console, there is also a getty running where you can log in as root and look around.

  6. To customize your build, you sooner or later want to add additional files to the image, for example some additional init scripts in the /etc/init.d/ folder. Or in the above case, it would be good to also have getty running on the graphical console. So to add custom files, the best way is to create an overlay folder which will be copied into the destination filesystem during the make process: 
    $ mkdir overlay/etc/init.d
$ cp my-startup-script.sh overlay/etc/init.d/S99myscript  # If you have one
$ cp output/target/etc/inittab overlay/etc/inittab
$ echo 'tty1::respawn:/sbin/getty -L tty1 0 linux' >> overlay/etc/inittab

    Then run make menuconfig and set the Root filesystem overlay directories option in the System Configuration section to the overlay folder that you have just created. Run make again and the next time you start your guest, you should see the new files in the image, e.g. also a getty running on the graphical console. Note: Do not try to add/change files directly in the output/target/ folder. That looks tempting first, but this is just a temporary folder used by the build system, which can be overwritten at any time and will be erased when you run make clean for example.

  7. If you need to tweak the kernel configuration, you can run make linux-menuconfig and do the appropriate changes there. For example, if you want to get keyboard input for the ppc64 pseries machine on the graphical console, you should enable the USB XHCI driver in the kernel, too. Once you are happy with the kernel configuration, save it, exit the menu and type make linux-rebuild && make. Note: To avoid that the kernel config gets reset after you run make clean at a later point in time, you should copy output/build/linux-*/.config to a safe location. Then run make menuconfig, change the Kernel -> Kernel configuration setting to Use a custom config file and set the Configuration file path to the copied file.

  8. If you want to add additional software to your image, you basically have to provide a Config.in file and a *.mk file. I recommend to have a look at the various packages in the package/ directory. Use one of the software from there with a similar build system as a template, and have a closer look at buildroot manual for details. Tweaking the build system of your software to properly cross-compile can be a little bit tricky some times, but most software that uses standard systems like autoconf should be fine.

That’s it. You now should be able to package your software in really small VM images. Of course, there are still lots of other settings that you can tweak in the buildroot environment – if you need any of these just have a look at the good buildroot manual for more information.