OpenShift Installation Guide (Agent-Based and Appliance)

This document covers the OpenShift installation process for bare metal and virtual machines using the agent-based installer and the appliance builder, targeting OpenShift 4.18 through 4.20.

1. OpenShift Appliance Image Provisioning

1.1 Overview

The openshift-appliance utility builds a self-contained disk image that orchestrates OpenShift installation using the agent-based installer. Its primary use case is fully disconnected (air-gapped) installations where no internet connectivity or external container registries are available.

All required container images (the OCP release payload, operators, additional images) are embedded directly in the disk image. The resulting image is generic: the same image can deploy multiple different OpenShift clusters. Cluster-specific configuration is provided separately via a config-image ISO at deployment time.

The utility is shipped as a container image: quay.io/edge-infrastructure/openshift-appliance

1.2 Appliance vs. Standard Agent-Based Installer

1.3 Build Process

The build has eight internal stages:

1. CoreOS ISO acquisition from the OCP release payload
2. Recovery ISO generation with custom bootstrap ignition
3. Registry image retrieval (for serving images during install)
4. Bootstrap-phase image collection (partial payload)
5. Installation-phase image retrieval (complete payload)
6. Data ISO assembly combining the registry and release images
7. Base CoreOS disk image download
8. Final appliance disk image construction (using guestfish)

Practical workflow:

export APPLIANCE_IMAGE="quay.io/edge-infrastructure/openshift-appliance"
export APPLIANCE_ASSETS="/absolute/path/to/assets"

# Generate configuration template
podman run --rm -it --pull newer \
  -v $APPLIANCE_ASSETS:/assets:Z \
  $APPLIANCE_IMAGE generate-config

# Edit appliance-config.yaml (see section 1.4)

# Build the disk image (requires privileged + host networking)
sudo podman run --rm -it --pull newer --privileged --net=host \
  -v $APPLIANCE_ASSETS:/assets:Z \
  $APPLIANCE_IMAGE build

The --privileged flag is required because guestfish is used to construct the disk image. The --net=host flag is required because an internal image registry container runs during the build. Disk space must be at least the configured diskSizeGB value (minimum 150 GiB).

1.4 appliance-config.yaml

apiVersion: v1beta1
kind: ApplianceConfig

Core fields:

Operators structure:

operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.19
    packages:
      - name: sriov-network-operator
        channels:
          - name: stable
      - name: local-storage-operator

1.5 Disk Image Layout

The resulting appliance.raw image contains:

1.6 Deployment (Two-Stage Process)

Factory stage – clone appliance.raw to target machines using dd, virt-resize, or the deployment ISO:

# Generate optional deployment ISO
podman run --rm -it --pull newer \
        -v $APPLIANCE_ASSETS:/assets:Z \
        $APPLIANCE_IMAGE deploy-iso-image

User site stage – provide cluster-specific configuration:

# Prepare install-config.yaml and agent-config.yaml
# Then generate the config-image ISO (NOT bootable, data only)
openshift-install agent create config-image --dir ./workdir

This produces agentconfig.noarch.iso. Mount it as a CD-ROM or USB on every node, then boot from the appliance disk. The appliance detects /media/config-image and begins installation automatically.

Recovery: reboot all nodes and select “Recovery: Agent-Based Installer” from the GRUB menu.

1.7 Recent Changes

• OCP 4.16: Appliance builder introduced as Technology Preview.
• OCP 4.17: Upgrade ISO workflow added (experimental).
• OCP 4.18: enableInteractiveFlow and useDefaultSourceNames options added.
• OCP 4.19: PinnedImageSets approaching GA; Nutanix platform support for agent-based installer.

1.8 References

• openshift/appliance GitHub repository
• openshift/appliance user guide
• Red Hat Customer Portal – Appliance Builder User Guide
• Appliance-based install notes (labguides)
• openshift-examples/openshift-appliance (KVM demo)

2. Agent-Based Installer Files and Their Purpose

2.1 install-config.yaml

This file defines the cluster topology, networking, platform, and credentials. All settings are install-time only and cannot be changed after installation.

Minimal Structure

apiVersion: v1
baseDomain: example.com
metadata:
  name: my-cluster
pullSecret: '{"auths": ...}'
sshKey: 'ssh-ed25519 AAAA...'

Networking

networking:
  networkType: OVNKubernetes  # Only supported type in 4.17+
  clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
  serviceNetwork:
    - 172.30.0.0/16
  machineNetwork:
    - cidr: 192.168.1.0/24

OpenShiftSDN was removed in OCP 4.17. Only OVNKubernetes is supported.

Compute and Control Plane Pools

compute:
  - name: worker
    replicas: 0       # 0 for SNO or compact 3-node
controlPlane:
  name: master
  replicas: 1         # 1 for SNO, 3 for HA

Platform Types

platform: none – no platform integration. User must provide external DNS and load balancing. Supported only for SNO in OCP 4.18+.

platform:
  none: {}

platform: baremetal – manages VIPs via keepalived/haproxy. Required for multi-node clusters:

platform:
  baremetal:
    apiVIPs:
      - 192.168.1.10
    ingressVIPs:
      - 192.168.1.11

platform: vsphere – VMware vSphere with vCenter integration:

platform:
  vsphere:
    apiVIPs:
      - 192.168.1.10
    ingressVIPs:
      - 192.168.1.11
    vCenter: vcenter.example.com
    username: administrator@vsphere.local
    password: secret
    datacenter: dc1
    defaultDatastore: datastore1

Proxy Settings

proxy:
  httpProxy: http://proxy.example.com:3128
  httpsProxy: http://proxy.example.com:3128
  noProxy: .example.com,10.0.0.0/8,172.16.0.0/12

Image Mirror / Disconnected Registry

imageContentSources is deprecated since OCP 4.14. Use imageDigestSources instead, which generates ImageDigestMirrorSet (IDMS) resources:

imageDigestSources:
  - mirrors:
      - mirror.example.com:5000/ocp4/openshift4
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  - mirrors:
      - mirror.example.com:5000/ocp4/openshift4
    source: quay.io/openshift-release-dev/ocp-release

Additional Trust Bundle

additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  <PEM-encoded CA certificate(s)>
  -----END CERTIFICATE-----

FIPS

fips: true   # Must be set at install time; cannot be changed later

Capabilities (4.14+)

capabilities:
  baselineCapabilitySet: None
  additionalEnabledCapabilities:
    - marketplace
    - openshift-samples

2.2 agent-config.yaml

This file defines host-specific settings: rendezvous IP, per-host network configuration, root device hints, interfaces, and roles.

API Version History

• v1alpha1 – original version, used from OCP 4.12. Still accepted.
• v1beta1 – introduced around OCP 4.14, signals schema stabilization.

Full Structure

apiVersion: v1beta1
kind: AgentConfig
metadata:
  name: my-cluster               # Must match install-config.yaml
rendezvousIP: 192.168.1.10       # IP of one master node (bootstrap host)
bootArtifactsBaseURL: http://192.168.1.1:8080/boot-artifacts
                                 # Optional. Base URL for PXE boot artifacts.
hosts:
  - hostname: master-0
    role: master                 # "master" or "worker"
    interfaces:
      - name: eno1
        macAddress: 00:11:22:33:44:55
    rootDeviceHints:
      deviceName: /dev/sda
    networkConfig:               # NMState format
      interfaces:
        - name: eno1
          type: ethernet
          state: up
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 192.168.1.10
                prefix-length: 24
          ipv6:
            enabled: false
      dns-resolver:
        config:
          server:
            - 192.168.1.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.1.1
            next-hop-interface: eno1
            table-id: 254

rootDeviceHints

These come from the Metal3 BareMetalHost API. Multiple hints can be combined (the device must match all of them).

/dev/sdX names can change between boots on multi-disk systems. Prefer persistent identifiers like wwn or /dev/disk/by-path/....

NMState networkConfig Examples

Bond:

networkConfig:
  interfaces:
    - name: eno1
      type: ethernet
      state: up
    - name: eno2
      type: ethernet
      state: up
    - name: bond0
      type: bond
      state: up
      ipv4:
        enabled: true
        dhcp: false
        address:
          - ip: 192.168.1.11
            prefix-length: 24
      link-aggregation:
        mode: active-backup
        port:
          - eno1
          - eno2

VLAN:

networkConfig:
  interfaces:
    - name: eno1
      type: ethernet
      state: up
      ipv4:
        enabled: false
    - name: eno1.100
      type: vlan
      state: up
      vlan:
        base-iface: eno1
        id: 100
      ipv4:
        enabled: true
        dhcp: false
        address:
          - ip: 192.168.100.11
            prefix-length: 24

Host Count Constraint

The number of hosts in agent-config.yaml must not exceed the sum of compute.replicas + controlPlane.replicas from install-config.yaml.

2.3 MachineConfig and Butane Files

Ignition, Butane, and MachineConfig

Ignition is the low-level JSON configuration format used by RHCOS to provision nodes. It handles file creation, systemd unit setup, disk partitioning, etc.

Butane is a human-friendly YAML transpiler that converts readable .bu files into either raw Ignition JSON or OpenShift MachineConfig CRDs. By default (without -r), butane with variant: openshift produces a MachineConfig YAML.

MachineConfig is an OpenShift CRD (machineconfiguration.openshift.io/v1) that wraps an Ignition config and targets a Machine Config Pool (typically master or worker). The Machine Config Operator (MCO) applies MachineConfigs to nodes.

Butane Config Structure (v4.20)

variant: openshift
version: 4.20.0
metadata:
  name: 99-worker-custom-config
  labels:
    machineconfiguration.openshift.io/role: worker
storage:
  files:
    - path: /etc/my-config
      mode: 0644
      contents:
        inline: |
          some configuration content
    - path: /usr/local/bin/my-script
      mode: 0755
      contents:
        local: my-script           # resolved relative to --files-dir
systemd:
  units:
    - name: my-service.service
      enabled: true
      contents: |
        [Unit]
        Description=My Custom Service
        [Service]
        ExecStart=/usr/local/bin/my-script
        [Install]
        WantedBy=multi-user.target
openshift:
  kernel_arguments:
    - hugepagesz=1G
    - hugepages=4
  fips: true
  kernel_type: default             # or "realtime"

The metadata.labels field with machineconfiguration.openshift.io/role: master or worker determines which MachineConfigPool the config targets.

The openshift section supports:

• kernel_type: default or realtime
• kernel_arguments: kernel command-line parameters
• extensions: RHCOS extensions to install
• fips: enable FIPS 140-2

Transpiling Butane to MachineConfig

butane my-config.bu --files-dir ./files -o openshift/99-my-config.yaml

The output is a MachineConfig CRD YAML with file contents base64-encoded in the spec.config Ignition payload.

Storage Section Details

Files can source content from:

• inline: direct text content
• local: path relative to --files-dir argument
• source: remote URL (http, https, data URI)

Directories and trees can be embedded. Filesystems, RAID, and LUKS encryption are also configurable.

Machine Config Operator (MCO)

The MCO has two components:

• Machine Config Server (MCS): provides Ignition files over HTTPS during bootstrap.
• Machine Config Daemon (MCD): runs on each node, detects drift, and applies changes.

When a MachineConfig is created or modified post-install, the MCO renders all MachineConfigs for a pool into a single config, then cordons, drains, applies, reboots, and uncordons nodes one at a time.

Day-1 vs Day-2

• Day-1 (install time): place MachineConfig YAML files in <install_dir>/openshift/ before running openshift-install agent create image. They are baked into the ISO and applied during bootstrap.
• Day-2 (post-install): apply MachineConfig objects with oc apply -f from the admin workstation (the machine where openshift-install was run and where auth/kubeconfig was generated – see section 3.14). The MCO detects the change and rolls it out with rolling reboots to the cluster nodes.

Example: Day-0 Services with Podman Quadlets

A common use case is deploying base infrastructure services that must be running before OpenShift itself starts (i.e. before kubelet and CRI-O). These services run as systemd-managed Podman containers (called “quadlets”) directly on the RHCOS host, outside of Kubernetes.

The key mechanism is systemd ordering: quadlet units specify Before=kubelet.service crio.service so that systemd starts them before the OpenShift node services. This is useful for DPDK dataplanes, routing daemons, or any infrastructure that must own hardware resources before Kubernetes takes over.

How Podman quadlets work: RHCOS includes a systemd generator that reads .container, .pod, and .volume files from /etc/containers/systemd/ and generates corresponding systemd units at boot. No extra tooling is required.

The following real-world example deploys a DPDK dataplane (grout), a routing suite (FRR), and a metrics exporter as a pod of three containers.

Butane file (30-hbn-quadlets.bu):

variant: openshift
version: 4.19.0
metadata:
  name: 99-hbn-quadlets
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
    # Helper script to bind NICs to vfio-pci (or move mlx5 to a netns)
    - path: /usr/bin/grout-bind
      mode: 0755
      contents:
        local: grout-bind
    # Systemd template unit: grout-bind@<netdev>.service
    - path: /etc/systemd/system/grout-bind@.service
      mode: 0644
      contents:
        local: grout-bind@.service
    # Quadlet definitions (pod + volume + containers)
    - path: /etc/containers/systemd/hbn.pod
      mode: 0644
      contents:
        local: hbn.pod
    - path: /etc/containers/systemd/hbn-run.volume
      mode: 0644
      contents:
        local: hbn-run.volume
    - path: /etc/containers/systemd/grout.container
      mode: 0644
      contents:
        local: grout.container
    - path: /etc/containers/systemd/grout-frr.container
      mode: 0644
      contents:
        local: grout-frr.container
    - path: /etc/containers/systemd/grout-metrics.container
      mode: 0644
      contents:
        local: grout-metrics.container
    # Configuration files
    - path: /etc/grout/interfaces
      mode: 0644
      contents:
        local: grout-interfaces
    - path: /etc/frr/daemons
      mode: 0644
      contents:
        local: frr-daemons
    - path: /etc/frr/frr.conf
      mode: 0644
      contents:
        local: frr.conf

Each local: reference is a file resolved relative to the --files-dir argument passed to butane. Here is what the referenced quadlet files contain:

Pod definition (hbn.pod) – groups all containers in a shared network and PID namespace with a shared /run volume:

[Pod]
PodName=hbn
Network=none
Volume=hbn-run.volume:/run

[Install]
WantedBy=multi-user.target default.target

Main container (grout.container) – the DPDK dataplane. Note the Before= lines that ensure it starts before kubelet and CRI-O:

[Unit]
Description=GROUT DPDK dataplane
After=dev-hugepages.mount
Before=kubelet.service crio.service ovs-configuration.service
Requires=dev-hugepages.mount

[Container]
Image=quay.io/grout/grout:edge
ContainerName=grout
Pod=hbn.pod
Notify=true
Exec=/usr/bin/grout -m 0666 -M unix:/run/grout-metrics.sock
PodmanArgs=--privileged --cpuset-cpus 0,2,3
Volume=/dev/hugepages:/dev/hugepages
Volume=/dev/vfio:/dev/vfio
Volume=/sys/bus/pci:/sys/bus/pci
Volume=/etc/grout:/etc/grout:ro

[Service]
ExecStartPre=/usr/bin/udevadm settle
ExecStartPost=/usr/bin/podman exec grout /usr/bin/grcli -xef /etc/grout/interfaces
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target default.target

Sidecar container (grout-frr.container) – a routing daemon that depends on the dataplane:

[Unit]
Description=FRR routing suite
After=grout.service
Before=kubelet.service crio.service ovs-configuration.service
Requires=grout.service

[Container]
Image=quay.io/grout/frr:edge
ContainerName=grout-frr
Pod=hbn.pod
PodmanArgs=--privileged --cpuset-cpus 0,1,2
Volume=/etc/frr:/etc/frr

[Service]
Type=forking
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target default.target

Transpile and include in the install directory:

mkdir -p install/openshift
butane machine_configs/30-hbn-quadlets.bu \
        --files-dir machine_configs \
        -o install/openshift/99-hbn-quadlets.yaml

butane reads all local: file references from machine_configs/, base64-encodes them, and embeds everything into a single MachineConfig YAML. When placed in install/openshift/ before ISO generation, the MCO applies this config during the first boot. The systemd ordering guarantees the following boot sequence:

1. grout-bind@<nic>.service – binds NICs to vfio-pci or moves them to a network namespace
2. grout.service (from grout.container) – starts the DPDK dataplane
3. grout-frr.service (from grout-frr.container) – starts FRR routing
4. kubelet.service / crio.service – OpenShift node services start after the dataplane is ready

The container images must be available at boot time. For connected installs they are pulled from the registry. For disconnected/appliance installs, list them in appliance-config.yaml under additionalImages so they are embedded in the disk image.

2.4 The Manifests Directory

The install directory layout before ISO generation:

<install_dir>/
  install-config.yaml
  agent-config.yaml
  openshift/                      # Extra manifests go here
    99-worker-custom.yaml         # MachineConfig or other CRDs
    99-master-kernel-args.yaml
    my-namespace.yaml

Any valid Kubernetes/OpenShift manifest placed in openshift/ will be applied during cluster bootstrap. The installer does not validate their contents beyond basic YAML parsing.

Optionally, openshift-install agent create cluster-manifests --dir <dir> can be used to generate and inspect cluster manifests before creating the image.

2.5 References

• OCP 4.20 Agent-Based Installer (full documentation)
• OCP 4.18 Installation Config Parameters
• OCP 4.19 Agent-Based Installer
• Butane OpenShift spec v4.20
• Butane OpenShift spec v4.18
• openshift/installer customization docs
• openshift/machine-config-operator
• Metal3 root device hints
• assisted-service install customization

3. openshift-install Flow on Bare Metal and Virtual Machines

3.1 Prerequisites

All tools listed below are installed on the admin workstation (e.g. your Fedora laptop), not on the cluster nodes.

• openshift-install binary matching the target OCP version
• oc CLI client (the OpenShift command-line tool)
• butane for transpiling .bu files to MachineConfig YAML
• nmstatectl for validating network configuration
• Pull secret from console.redhat.com

Installing the Tools on Fedora

oc and openshift-install are not packaged in Fedora repositories. They must be downloaded from Red Hat as pre-built binaries:

# Pick the OCP version you want to install
OCP_VERSION=4.19.0
BASE_URL=https://mirror.openshift.com/pub/openshift-v4/clients/ocp

# Download and extract oc + kubectl
curl -sL $BASE_URL/$OCP_VERSION/openshift-client-linux.tar.gz \
       | tar xz -C ~/.local/bin oc kubectl

# Download and extract openshift-install
curl -sL $BASE_URL/$OCP_VERSION/openshift-install-linux.tar.gz \
       | tar xz -C ~/.local/bin openshift-install

# Verify
oc version --client
openshift-install version

Alternatively, oc can be extracted from the release image itself:

oc adm release extract --tools quay.io/openshift-release-dev/ocp-release:${OCP_VERSION}-x86_64

butane is packaged in Fedora:

sudo dnf install butane

nmstatectl is also packaged in Fedora:

sudo dnf install nmstate

3.2 Preparation

mkdir install
cp install-config.yaml agent-config.yaml install/

# Transpile any Butane configs into MachineConfig manifests
mkdir -p install/openshift
butane machine_configs/99-custom.bu --files-dir ./files \
         -o install/openshift/99-custom.yaml

# Optionally inspect generated cluster manifests
openshift-install agent create cluster-manifests --dir install

WARNING: openshift-install agent create image and openshift-install agent create config-image destroy the input configuration files (install-config.yaml, agent-config.yaml). Always work from copies.

3.3 ISO Generation

Three subcommands are available:

# Standard install
openshift-install agent create image --dir=install --log-level=debug

# Appliance workflow
openshift-install agent create config-image --dir=install

# PXE boot
openshift-install agent create pxe-files --dir=install

For PXE boot, set bootArtifactsBaseURL in agent-config.yaml to the HTTP server URL hosting the generated files.

3.4 Booting Nodes

Boot all target machines from the ISO using whichever method is available:

• USB drive
• Virtual media (Redfish/iDRAC/iLO)
• CD-ROM mount (VMs)
• PXE/iPXE network boot

For the appliance workflow, write appliance.raw to disk and mount the config-image ISO as a CD-ROM or USB, then boot from disk.

3.5 The Bootstrap Process (Internal Flow)

The agent ISO contains two key components:

1. Assisted discovery agent – runs on every node
2. Assisted Service – runs only on the rendezvous host

The internal sequence:

1. ISO Boot: all nodes boot from the agent ISO into a RHCOS live environment.

2. Rendezvous host selection: the node whose IP matches rendezvousIP in agent-config.yaml starts the Assisted Service. This node acts as the bootstrap host.

3. Agent discovery: each host’s agent contacts the Assisted Service via REST API, sending hardware inventory data (CPU, memory, disks, NICs) and connectivity information.

4. Host validation: the Assisted Service validates all hosts:

   • Connectivity between hosts
   • NTP synchronization
   • DNS resolution (api.<cluster>.<domain>, api-int.<cluster>.<domain>, *.apps.<cluster>.<domain>)
   • Hardware requirements (minimum CPU, RAM, disk)
   • Network requirements (MTU checks as of 4.19)

5. Install trigger: once all expected hosts are discovered and validated, the Assisted Service automatically triggers installation.

6. RHCOS write: all nodes have the RHCOS image written to their disks.

7. Non-bootstrap reboot: non-bootstrap nodes reboot first and begin forming the cluster (etcd, API server, etc.).

8. Bootstrap reboot: the rendezvous host reboots last and joins the cluster as a regular control plane node. No separate bootstrap machine is needed.

9. Cluster operator deployment: cluster operators come online progressively. MachineConfigs from the openshift/ directory are applied by the MCO.

3.6 The Rendezvous Host

The rendezvous host is the control plane node that runs the Assisted Service during bootstrap:

• Identified by rendezvousIP in agent-config.yaml.
• Must be a control plane (master) node.
• After bootstrapping completes, it reboots and joins the cluster as a normal control plane node.
• Unlike IPI bare metal installation, no separate provisioning machine is required.

3.7 Bare Metal Specifics

BMC Configuration

BMC fields in install-config.yaml are optional for the agent-based installer (the user boots nodes manually from the ISO). However, when using platform: baremetal, the BMC fields enable post-install bare metal management via Metal3/BMO.

Supported BMC address formats:

• IPMI: ipmi://<ip>:<port>
• Redfish: redfish://<ip>/redfish/v1/Systems/1
• Redfish virtual media: redfish-virtualmedia://<ip>/redfish/v1/Systems/1
• Dell iDRAC: idrac-virtualmedia://<ip>/redfish/v1/Systems/System.Embedded.1

DNS Requirements

The following DNS records must be configured before installation:

For SNO, all three records should point to the single node’s IP.

3.8 Virtual Machine Specifics

VMs on KVM/libvirt are treated as “bare metal” from the installer’s perspective. MAC addresses must be defined before generating the ISO.

Example using virt-install:

virt-install \
       --name ocp-sno \
       --memory 65536 \
       --vcpus 12 \
       --os-variant fedora-coreos-stable \
       --cpu host-passthrough \
       --disk path=/var/lib/libvirt/images/ocp-sno.qcow2,size=120 \
       --network network=ocp-net,mac=02:01:00:00:00:66 \
       --cdrom /path/to/agent.x86_64.iso

3.9 SNO (Single Node OpenShift)

SNO uses a single node as both master and worker. Key differences:

• controlPlane.replicas: 1, compute[0].replicas: 0
• platform: none or platform: baremetal with a single host
• In-place bootstrap: the single node is both the rendezvous host and the final control plane
• Worker nodes can be added later as day-2 operations

Example install-config.yaml for SNO:

apiVersion: v1
baseDomain: example.com
compute:
  - name: worker
    replicas: 0
controlPlane:
  name: master
  replicas: 1
metadata:
  name: sno-cluster
networking:
  clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
  machineNetwork:
    - cidr: 192.168.1.0/24
  networkType: OVNKubernetes
  serviceNetwork:
    - 172.30.0.0/16
platform:
  none: {}
pullSecret: '<pull_secret>'
sshKey: '<ssh_pub_key>'

3.10 Supported Topologies

3.11 Monitoring Installation

# Wait for bootstrap (control plane up, API available)
openshift-install agent wait-for bootstrap-complete \
        --dir=install --log-level=debug

# Wait for full installation (all operators ready)
openshift-install agent wait-for install-complete \
        --dir=install --log-level=debug

Debug output is also written to install/.openshift_install.log.

3.12 SSH Access During Installation

ssh core@<node-ip>

# Check bootkube progress
ssh core@<node-ip> journalctl -b -f -u bootkube.service

# Check assisted-service logs (rendezvous host)
ssh core@<node-ip> 'sudo podman logs assisted-service'

3.13 Troubleshooting

Bootstrap timeout (context deadline exceeded):

• Check DNS resolution for api.<cluster>.<domain> and api-int.<cluster>.<domain>.
• Check NTP synchronization across nodes.
• Verify network connectivity between all nodes.
• Verify firewall rules for ports 6443, 22623, 443, 80.
• For platform: baremetal, verify VIP addresses are in the machine network and not already in use.

Gather bootstrap logs:

openshift-install gather bootstrap \
        --dir=install \
        --bootstrap <bootstrap_ip> \
        --master <master1_ip> --master <master2_ip>

3.14 Post-Installation

The Admin Workstation

OpenShift has a client-server architecture. The cluster nodes run the OpenShift/Kubernetes API server and all workloads. To manage the cluster, you use the oc command-line client (analogous to kubectl in plain Kubernetes) from an admin workstation – any machine with network access to the cluster’s API endpoint (api.<cluster>.<domain>:6443). This is typically the same machine where openshift-install was run.

The oc client does not run on the cluster nodes themselves. It communicates with the cluster over HTTPS using a credentials file called kubeconfig. The installer generates this file during installation.

Credentials

After successful installation, credentials are at:

• kubeconfig: <install_dir>/auth/kubeconfig
• kubeadmin password: <install_dir>/auth/kubeadmin-password

All oc and openshift-install commands below are run on the admin workstation, not on the cluster nodes:

# Tell oc where to find the cluster credentials
export KUBECONFIG=~/install/auth/kubeconfig

# Verify connectivity and authentication
oc whoami                    # should print "system:admin"
oc get nodes                 # list cluster nodes and their status
oc get clusteroperators      # list all operators and their health

Any day-2 changes (applying MachineConfigs, creating namespaces, deploying workloads) are done the same way – by running oc commands from the admin workstation. The oc client sends the request to the cluster API, and the relevant controller running inside the cluster acts on it. For example:

# Apply a MachineConfig from the admin workstation
oc apply -f 99-custom-machineconfig.yaml

# The Machine Config Operator (running inside the cluster) picks up the
# change and rolls it out to the affected nodes automatically.

The wait-for install-complete command outputs the web console URL and kubeadmin credentials.

3.15 References

• OCP 4.20 Agent-Based Installer (full documentation)
• OCP 4.18 Agent-Based Installer
• OCP 4.18 Preparing for bare metal
• OCP 4.18 Installing on a single node
• OCP 4.20 Release Notes
• Red Hat Blog – Meet the New Agent-Based OpenShift Installer
• Red Hat Blog – PXE Booting with Agent-Based Installer
• Agent-Based Install Notes (labguides)
• Red Hat SE RTO – OpenShift Agent Install
• openshift/assisted-service

4. TL;DR – SNO in a VM from Scratch

This section walks through a complete example: building an appliance disk image, generating a config ISO, booting a SNO VM, and interacting with the resulting cluster. All commands are run on the admin workstation (your Fedora laptop) unless noted otherwise.

4.1 Install Tools and Set Up libvirt

Virtualization Packages

Install the KVM/libvirt/QEMU stack:

sudo dnf install -y \
        qemu-kvm \
        libvirt \
        libvirt-daemon-kvm \
        libvirt-daemon-config-network \
        libvirt-client \
        virt-install \
        guestfs-tools

sudo systemctl enable --now libvirtd
sudo usermod -aG libvirt,kvm $USER
# Log out and back in for group membership to take effect

Being in the libvirt and kvm groups allows running virsh, virt-install, and qemu-img without sudo.

Package breakdown:

• qemu-kvm – QEMU with KVM hardware acceleration
• libvirt, libvirt-daemon-kvm – virtualization management daemon
• libvirt-daemon-config-network – default NAT network setup
• libvirt-client – virsh command-line tool
• virt-install – command-line VM creation tool
• guestfs-tools – disk image manipulation (virt-resize, guestfish, etc.)

OpenShift and Butane Tools

sudo dnf install -y butane nmstate

OCP_VERSION=4.19.0
BASE_URL=https://mirror.openshift.com/pub/openshift-v4/clients/ocp

curl -sL $BASE_URL/$OCP_VERSION/openshift-client-linux.tar.gz \
       | tar xz -C ~/.local/bin oc kubectl

curl -sL $BASE_URL/$OCP_VERSION/openshift-install-linux.tar.gz \
       | tar xz -C ~/.local/bin openshift-install

Create the libvirt Network

Each libvirt virtual network automatically spawns its own dnsmasq instance that provides DHCP and DNS for VMs on that network. You do not install or configure dnsmasq directly – libvirt manages it from the network XML definition.

OpenShift requires three DNS records pointing to the SNO node’s IP. The native libvirt <dns><host> XML element does not support wildcard entries (*.apps...), so we must use the <dnsmasq:options> extension namespace to pass raw dnsmasq directives.

Create the network XML:

cat > /tmp/ocp-net.xml <<'EOF'
<network xmlns:dnsmasq="http://libvirt.org/schemas/network/dnsmasq/1.0">
  <name>ocp-net</name>
  <forward mode="nat"/>
  <bridge name="virbr-ocp" stp="on" delay="0"/>

  <ip address="192.168.113.1" netmask="255.255.255.0">
    <dhcp>
      <range start="192.168.113.2" end="192.168.113.49"/>
    </dhcp>
  </ip>

  <dnsmasq:options>
    <dnsmasq:option value="address=/api.sno.example.com/192.168.113.50"/>
    <dnsmasq:option value="address=/api-int.sno.example.com/192.168.113.50"/>
    <!-- Leading dot = wildcard: matches *.apps.sno.example.com -->
    <dnsmasq:option value="address=/.apps.sno.example.com/192.168.113.50"/>
  </dnsmasq:options>
</network>
EOF

Define, start, and autostart the network:

virsh net-define /tmp/ocp-net.xml
virsh net-start ocp-net
virsh net-autostart ocp-net

# Verify it is running
virsh net-list

You can inspect what libvirt generated for dnsmasq:

cat /var/lib/libvirt/dnsmasq/ocp-net.conf

Test that DNS works (queries go to the libvirt dnsmasq on the bridge IP):

dig @192.168.113.1 api.sno.example.com +short
# 192.168.113.50

dig @192.168.113.1 anything.apps.sno.example.com +short
# 192.168.113.50

Make the Host Resolve OpenShift DNS Names

The libvirt dnsmasq only listens on the bridge interface (192.168.113.1). Your admin workstation (the host itself) does not use it by default – tools like oc and openshift-install will fail to resolve api.sno.example.com unless you configure the host’s DNS.

Option A – systemd-resolved (recommended on Fedora):

Tell systemd-resolved to route queries for sno.example.com to the libvirt dnsmasq. The ~ prefix means “routing domain” – only matching queries are sent there, everything else uses your normal DNS.

sudo resolvectl dns virbr-ocp 192.168.113.1
sudo resolvectl domain virbr-ocp "~sno.example.com"

# Verify
resolvectl status virbr-ocp

This does not survive a network restart. To re-apply it automatically, create a libvirt hook script:

sudo mkdir -p /etc/libvirt/hooks
sudo tee /etc/libvirt/hooks/network <<'HOOK'
#!/bin/bash
# $1=network name, $2=action (started|stopped|...)
if [ "$1" = "ocp-net" ] && [ "$2" = "started" ]; then
        resolvectl dns virbr-ocp 192.168.113.1
        resolvectl domain virbr-ocp "~sno.example.com"
fi
HOOK
sudo chmod +x /etc/libvirt/hooks/network
sudo systemctl restart libvirtd

Option B – NetworkManager dnsmasq plugin:

This makes NetworkManager run its own local dnsmasq that forwards OpenShift queries to the libvirt dnsmasq. Persistent across reboots without hooks.

# Enable NetworkManager's dnsmasq DNS plugin
sudo tee /etc/NetworkManager/conf.d/00-use-dnsmasq.conf <<'EOF'
[main]
dns=dnsmasq
EOF

# Forward OpenShift queries to the libvirt bridge
sudo tee /etc/NetworkManager/dnsmasq.d/openshift-sno.conf <<'EOF'
server=/sno.example.com/192.168.113.1
EOF

sudo systemctl restart NetworkManager

Verify from the host (regardless of which option you chose):

dig api.sno.example.com +short
# 192.168.113.50

dig console-openshift-console.apps.sno.example.com +short
# 192.168.113.50

4.2 Build the Appliance Disk Image

Create a working directory and generate the config template:

mkdir -p ~/appliance-assets
export APPLIANCE_IMAGE="quay.io/edge-infrastructure/openshift-appliance"
export APPLIANCE_ASSETS=~/appliance-assets

podman run --rm -it --pull newer \
        -v $APPLIANCE_ASSETS:/assets:Z \
        $APPLIANCE_IMAGE generate-config

Edit ~/appliance-assets/appliance-config.yaml:

apiVersion: v1beta1
kind: ApplianceConfig
ocpRelease:
  version: "4.19"
  channel: stable
  cpuArchitecture: x86_64
diskSizeGB: 200
pullSecret: '<paste your pull secret from console.redhat.com>'
sshKey: '<paste your ~/.ssh/id_ed25519.pub>'
userCorePass: 'changeme'

Build (takes a long time – it downloads all OCP container images):

sudo podman run --rm -it --pull newer --privileged --net=host \
        -v $APPLIANCE_ASSETS:/assets:Z \
        $APPLIANCE_IMAGE build

The output is ~/appliance-assets/appliance.raw (~200 GB).

4.3 Prepare Cluster Configuration

Pick a cluster name, domain, and IP for your SNO node. This example uses the ocp-net libvirt network created in section 4.1 (192.168.113.0/24, DNS already configured).

Create a working directory with the two config files:

mkdir -p ~/sno-install

# ~/sno-install/install-config.yaml
apiVersion: v1
baseDomain: example.com
metadata:
  name: sno
compute:
  - name: worker
    replicas: 0
controlPlane:
  name: master
  replicas: 1
networking:
  networkType: OVNKubernetes
  clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
  serviceNetwork:
    - 172.30.0.0/16
  machineNetwork:
    - cidr: 192.168.113.0/24
platform:
  none: {}
pullSecret: '<paste your pull secret>'
sshKey: '<paste your ~/.ssh/id_ed25519.pub>'

# ~/sno-install/agent-config.yaml
apiVersion: v1beta1
kind: AgentConfig
metadata:
  name: sno
rendezvousIP: 192.168.113.50
hosts:
  - hostname: sno
    role: master
    interfaces:
      - name: enp1s0
        macAddress: "52:54:00:aa:bb:cc"
    rootDeviceHints:
      deviceName: /dev/vda
    networkConfig:
      interfaces:
        - name: enp1s0
          type: ethernet
          state: up
          mac-address: "52:54:00:aa:bb:cc"
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 192.168.113.50
                prefix-length: 24
          ipv6:
            enabled: false
      dns-resolver:
        config:
          server:
            - 192.168.113.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.113.1
            next-hop-interface: enp1s0
            table-id: 254

The MAC address (52:54:00:aa:bb:cc) must match what you assign to the VM later. Pick it now.

4.4 Generate the Config ISO

openshift-install agent create config-image --dir ~/sno-install

This produces ~/sno-install/agentconfig.noarch.iso and destroys install-config.yaml and agent-config.yaml (keep backups).

4.5 Create and Boot the VM

Convert the appliance raw image to qcow2 for the VM disk, then create the VM with the config ISO attached as a CD-ROM:

# Create a qcow2 copy of the appliance image for this VM
qemu-img convert -f raw -O qcow2 \
        ~/appliance-assets/appliance.raw \
        /var/lib/libvirt/images/sno.qcow2

# Create the VM
virt-install \
        --name sno \
        --memory 65536 \
        --vcpus 12 \
        --os-variant fedora-coreos-stable \
        --cpu host-passthrough \
        --disk /var/lib/libvirt/images/sno.qcow2 \
        --network network=ocp-net,mac=52:54:00:aa:bb:cc \
        --disk ~/sno-install/agentconfig.noarch.iso,device=cdrom \
        --boot hd \
        --noautoconsole

The VM boots from the appliance disk, detects the config ISO, and starts the agent-based installer automatically.

4.6 Monitor the Installation

# Watch bootstrap progress (takes ~30 minutes)
openshift-install agent wait-for bootstrap-complete \
        --dir ~/sno-install --log-level=debug

# Watch full installation (takes another ~20 minutes after bootstrap)
openshift-install agent wait-for install-complete \
        --dir ~/sno-install --log-level=debug

You can also SSH into the node during installation to check progress:

ssh core@192.168.113.50 journalctl -b -f -u bootkube.service

4.7 Interact with the Cluster

Once wait-for install-complete succeeds, set up your shell:

export KUBECONFIG=~/sno-install/auth/kubeconfig

Check cluster health:

# Who am I?
oc whoami
# system:admin

# List nodes (should show one node with "Ready" status)
oc get nodes

# Check all cluster operators are Available
oc get clusteroperators

# Short alias: co
oc get co

Explore what is running:

# List all namespaces (an OpenShift cluster has many system namespaces)
oc get namespaces

# List all pods in all namespaces
oc get pods -A

# List pods in a specific namespace
oc get pods -n openshift-ovn-kubernetes

# Describe a pod for detailed info
oc describe pod <pod-name> -n <namespace>

# View logs of a pod
oc logs <pod-name> -n <namespace>

# Follow logs in real time
oc logs -f <pod-name> -n <namespace>

Deploy a test workload:

# Create a new project (OpenShift's term for namespace)
oc new-project test

# Run a simple container
oc run hello --image=registry.access.redhat.com/ubi9/ubi-minimal \
        --command -- sleep infinity

# Check it is running
oc get pods

# Execute a command inside the container
oc exec hello -- cat /etc/os-release

# Delete it
oc delete pod hello

# Delete the project
oc delete project test

Apply a day-2 MachineConfig:

# Transpile a Butane config on your workstation
butane 99-custom.bu --files-dir ./files -o 99-custom.yaml

# Send it to the cluster
oc apply -f 99-custom.yaml

# The MCO will drain the node, apply the config, reboot, and uncordon.
# Watch the MachineConfigPool progress:
oc get mcp

Access the web console:

The web console URL is printed by wait-for install-complete. It is typically https://console-openshift-console.apps.sno.example.com. Log in with username kubeadmin and the password from ~/sno-install/auth/kubeadmin-password.

SSH into the node:

ssh core@192.168.113.50

The core user has passwordless sudo. This is useful for debugging node-level issues but should not be used for day-to-day cluster management (use oc instead).