IPoIB: openib opensm: build order build and install libibverbs{,-devel}, build and install libmthca, build and install libibcommon{,-devel}, build and install libibumad{,-devel}, build and install libibmad{,-devel}, build and install opensm{,-libs} BTW, Fedora already has libibverbs, libmthca install libibverbs-utils service openibd start - Making the ib stack fully functional takes all those modules (needs to happen all machines) /etc/ofed/openib.conf controls what is loaded... don't forget to chkconfig --level 2345 {openibd,opensm} on dledford> when running ibv_devinfo, the sm lid: item is the lid of opensm, it should be the same on all machines the port_lid: is this particular port' s actual link id that's what you can use to talk between ports aka, the first machine you ran ibv_rc_pingpong on went into server mode and attached itself to local lid 1, so on the other machine you would tell it to run ibv_rc_pingpong and tell it to connect to lid 1 and it would resolve out to the first machine IB also has it's own port namespace, just like tcp/ip all the various commands default to certain ports, and you can use the same port on an IB fabric simultaneously with that port on tcp and they don't collide (of course, a given port over ipoib will collide with tcp/udp port space, but underneath ipoib the rdma layer is using a single port for all the ipoib traffic) you should download, compile, and install the perftest rpm ftp://ftp.nluug.nl/pub/os/Linux/distr/startcom/AS-4.0.0/updates/x86_64/perftest-1.0-1.x86_64.rpm eg: ib_send_bw -a will test all the normal send sizes, adding -b will test them bidirectionally a 10GBit/s fabric is good for 700/800MByte/s unidirectional, and 1400/1500 bidirectional, where as a 20GBit/s fabric usually gets 1300 uni and 2500 or so bi.