Use of IB Drivers Co-resident with OPA on Debian Jessie¶

The Omni-Path hardware and drivers should operate correctly when co-resident with InfiniBand HCAs and their associated drivers. However, on Jessie (Debian 8.x), kmod-ifs-kernel-updates installs a version of ib_mad that includes jumbo MAD support. This version of ib_mad is required for OPA, and is incompatible with the InfiniBand drivers that are in-box with Jessie.

This document is only relevant to Jessie. Debian Stretch's 4.9 kernel ib_mad module already includes support for jumbo MAD, and IB drivers should not require recompilation to be co-resident with OPA on that version of the operating system.

Current Behavior¶

The behavior that will be experienced when loading an in-box InfiniBand driver co-resident with OPA is a report of "invalid parameters". Syslog will contain reports of "disagrees about version of symbol ib_unregister_mad_agent", etc. This is due to the fact that the updated ib_mad module has a different version of its ABI than what is expected by the in-box drivers.

It is important to note that the InfiniBand SM must not be running on a machine that has OPA installed. The SM must be executed on an IB-only node. Furthermore, the OPA and IB fabrics are orthogonal to each other. There is no direct communication between the two types of fabric.

Building IB Drivers Against OPA-enabled Headers¶

In order to use the in-box InfiniBand drivers with the jumbo-MAD-enabled ib_mad module, they must be recompiled against the updated headers. This process is outlined below. The build system must have build-essentials installed. The build should be executed as a non-root user.

This process assumes that the mlx4 driver is to be included with kmod-ifs-kernel-updates. It should apply to other InfiniBand drivers.

# mkdir ifsbuild
# cd ifsbuild
# dget http://security.debian.org/debian-security/pool/updates/main/l/linux/linux_3.16.43-2+deb8u3.dsc
# cp ../IntelOPA-IFS.DEBIAN8-x86_64.10.4.2.0.7-2/packages/ifs-kernel-updates*{orig,debian}* .
# tar xf ifs-kernel-updates_3.16.0-616.orig.tar.gz
# cd ifs-kernel-updates-3.10.0_327.el7.x86_64
# tar xf ../ifs-kernel-updates_3.16.0-616-1ifs.debian.tar.xz
# QUILT_PATCHES=“debian/patches" quilt push -a
# cp -r ../linux-3.16.43/drivers/infiniband/hw/mlx4/ .
# cd mlx4
# mv Makefile Makefile.org
# cd ..

mlx4/Makefile must now be created. The source below will work for the current version of the mlx4 driver (Jessie 8.9). Makefile.org should be examined to determine the correct directives for obj-$(CONFIG_MLX4_INFINIBAND) and mlx4_ib-y.

#
# mlx4 module
#
#
# Called from the kernel module build system.
#
ifneq ($(KERNELRELEASE),)
#kbuild part of makefile

ccflags-y := -I$(src)/../include

obj-$(CONFIG_MLX4_INFINIBAND)   += mlx4_ib.o

mlx4_ib-y :=    ah.o cq.o doorbell.o mad.o main.o mr.o qp.o srq.o mcg.o cm.o alias_GUID.o sysfs.o

else
#normal makefile
KDIR ?= /lib/modules/`uname -r`/build

default:
    $(MAKE) -C $(KDIR) M=$$PWD NOSTDINC_FLAGS=-I$$PWD

clean:
    $(MAKE) -C $(KDIR) M=$$PWD clean

install:
    $(MAKE) INSTALL_MOD_DIR=updates -C $(KDIR) M=$$PWD modules_install

endif

Now, the top-level Makefile should be edited to include mlx4 in the build. Add mlx4/ to the end of the obj-y directive. The trailing slash is necessary.

obj-y := rdmavt/ ib_uverbs/ hfi1/ ib_mad/ opa_compat/ \
        ib_umad/ ib_sa/ ib_cm/ ib_ucm/ rdma_cm/ rdma_ucm/ ib_srpt/ ib_ipoib/ mlx4/

It may be desirable to update the package version in debian/changelog to indicate that the package is different than what was shipped with IFS. An entry may be made at the beginning of the file:

ifs-kernel-updates (3.16.0-616-1ifs-mlx4) UNRELEASED; urgency=medium

    * Add mlx4 module

 -- Your Name <youremail>  Tue, 11 Jul 2017 11:38:22 -0500

The package may now be built via the command:

# debuild -us -uc -b

The -b flag is necessary because the source does not include a patch to account for the mlx4 additions. If a source package is desired: exclude the -b flag, create a patch with quilt and add Makefile and mlx4/* to the patch. Further explanation of this process is beyond the scope of this document. See https://wiki.debian.org/UsingQuilt for more information.

The build should complete without errors, although lintian may complain about a few things. The package may now be installed with:

# sudo dpkg -i ../kmod-ifs-kernel-updates_3.16.0-616-1ifs_amd64.deb
# depmod -a

The system should now have a recompiled driver installed to /lib/modules/3.16.0-4-amd64/updates/ifs-kernel-updates/mlx4/mlx4_ib.ko and modprobe mlx4_ib should execute without errors. ibstat should include information for the mlx4_0 device.

The system will also need the libmlx4 package installed in order to provide user-space ibverbs access. User-space packages should not require recompilation.

Brian T. Smith
Senior Technical Staff
System Fabric Works, Inc.
bsmith@systemfabricworks.com