Follows are some notes from hard trial-and-error

… for getting SRIOV working with mlx4 and mlx5 drivers on linux for both IB and RoCE on Mellanox SX6012 and SX1012 switches.

mlx5 IB

Used a setup script

echo 2 > /sys/class/infiniband/mlx5_1/device/sriov_numvfs echo 2 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs

# Here’s how we set up stable port_guids and mac addrs for # the VFs we give to our guests for mlx5. The mlx4 settings are in # /etc/rdma/sriov-vfs # # for the rhel8 guest: ip link set mlx5_ib0 vf 0 node_guid 49:2f:7f:d1:b9:80:45:b9 ip link set mlx5_ib0 vf 0 port_guid 49:2f:7f:d1:b9:80:45:b8 ip link set mlx5_ib0 vf 0 state auto

ConnectX-4 card settings:

SRIOV_EN=1 FPP_EN=1 MULTI_PORT_VHCA_EN=1 SRIOV_IB_ROUTING_MODE_P2=0 SRIOV_IB_ROUTING_MODE_P1=0

On the Mellanox IB switch: switch-acce0c [standalone: master] (config) # ib sm virt enable