Use of IB Drivers Co-resident with OPA on Debian Jessie

The Omni-Path hardware and drivers should operate correctly when co-resident with InfiniBand HCAs and their associated drivers. However, on Jessie (Debian 8.x), kmod-ifs-kernel-updates installs a version of ib_mad that includes jumbo MAD support. This version of ib_mad is required for OPA, and is incompatible with the InfiniBand drivers that are in-box with Jessie.

This document is only relevant to Jessie. Debian Stretch's 4.9 kernel ib_mad module already includes support for jumbo MAD, and IB drivers should not require recompilation to be co-resident with OPA on that version of the operating system.

Current Behavior

The behavior that will be experienced when loading an in-box InfiniBand driver co-resident with OPA is a report of "invalid parameters". Syslog will contain reports of "disagrees about version of symbol ib_unregister_mad_agent", etc. This is due to the fact that the updated ib_mad module has a different version of its ABI than what is expected by the in-box drivers.

It is important to note that the InfiniBand SM must not be running on a machine that has OPA installed. The SM must be executed on an IB-only node. Furthermore, the OPA and IB fabrics are orthogonal to each other. There is no direct communication between the two types of fabric.

Building IB Drivers Against OPA-enabled Headers

In order to use the in-box InfiniBand drivers with the jumbo-MAD-enabled ib_mad module, they must be recompiled against the updated headers. This process is outlined below. The build system must have build-essentials installed. The build should be executed as a non-root user.

This process assumes that the mlx4 driver is to be included with kmod-ifs-kernel-updates. It should apply to other InfiniBand drivers.

# mkdir ifsbuild
# cd ifsbuild
# dget
# cp ../IntelOPA-IFS.DEBIAN8-x86_64.*{orig,debian}* .
# tar xf ifs-kernel-updates_3.16.0-616.orig.tar.gz
# cd ifs-kernel-updates-3.10.0_327.el7.x86_64
# tar xf ../ifs-kernel-updates_3.16.0-616-1ifs.debian.tar.xz
# QUILT_PATCHES=“debian/patches" quilt push -a
# cp -r ../linux-3.16.43/drivers/infiniband/hw/mlx4/ .
# cd mlx4
# mv Makefile
# cd ..

mlx4/Makefile must now be created. The source below will work for the current version of the mlx4 driver (Jessie 8.9). should be examined to determine the correct directives for obj-$(CONFIG_MLX4_INFINIBAND) and mlx4_ib-y.

# mlx4 module
# Called from the kernel module build system.
#kbuild part of makefile

ccflags-y := -I$(src)/../include

obj-$(CONFIG_MLX4_INFINIBAND)   += mlx4_ib.o

mlx4_ib-y :=    ah.o cq.o doorbell.o mad.o main.o mr.o qp.o srq.o mcg.o cm.o alias_GUID.o sysfs.o

#normal makefile
KDIR ?= /lib/modules/`uname -r`/build


    $(MAKE) -C $(KDIR) M=$$PWD clean

    $(MAKE) INSTALL_MOD_DIR=updates -C $(KDIR) M=$$PWD modules_install


Now, the top-level Makefile should be edited to include mlx4 in the build. Add mlx4/ to the end of the obj-y directive. The trailing slash is necessary.

obj-y := rdmavt/ ib_uverbs/ hfi1/ ib_mad/ opa_compat/ \
        ib_umad/ ib_sa/ ib_cm/ ib_ucm/ rdma_cm/ rdma_ucm/ ib_srpt/ ib_ipoib/ mlx4/

It may be desirable to update the package version in debian/changelog to indicate that the package is different than what was shipped with IFS. An entry may be made at the beginning of the file:

ifs-kernel-updates (3.16.0-616-1ifs-mlx4) UNRELEASED; urgency=medium

    * Add mlx4 module

 -- Your Name <youremail>  Tue, 11 Jul 2017 11:38:22 -0500

The package may now be built via the command:

# debuild -us -uc -b

The -b flag is necessary because the source does not include a patch to account for the mlx4 additions. If a source package is desired: exclude the -b flag, create a patch with quilt and add Makefile and mlx4/* to the patch. Further explanation of this process is beyond the scope of this document. See for more information.

The build should complete without errors, although lintian may complain about a few things. The package may now be installed with:

# sudo dpkg -i ../kmod-ifs-kernel-updates_3.16.0-616-1ifs_amd64.deb
# depmod -a

The system should now have a recompiled driver installed to /lib/modules/3.16.0-4-amd64/updates/ifs-kernel-updates/mlx4/mlx4_ib.ko and modprobe mlx4_ib should execute without errors. ibstat should include information for the mlx4_0 device.

The system will also need the libmlx4 package installed in order to provide user-space ibverbs access. User-space packages should not require recompilation.

Brian T. Smith
Senior Technical Staff
System Fabric Works, Inc.