Use of IB Drivers Co-resident with OPA on Debian Jessie » History » Version 15
Brian Smith, 08/31/2017 08:33 PM
1 | 15 | Brian Smith | # Use of IB Drivers Co-resident with OPA on Debian Jessie |
---|---|---|---|
2 | 1 | Brian Smith | |
3 | The Omni-Path hardware and drivers should operate correctly when co-resident with InfiniBand HCAs and their associated drivers. However, on Jessie (Debian 8.x), kmod-ifs-kernel-updates installs a version of ib_mad that includes jumbo MAD support. This version of ib_mad is required for OPA, and is incompatible with the InfiniBand drivers that are in-box with Jessie. |
||
4 | |||
5 | 14 | Brian Smith | This document is only relevant to Jessie. Debian Stretch's 4.9 kernel ib_mad module already includes support for jumbo MAD, and IB drivers should not require recompilation to be co-resident with OPA on that version of the operating system. |
6 | 12 | Brian Smith | |
7 | 8 | Brian Smith | ## Current Behavior |
8 | |||
9 | 1 | Brian Smith | The behavior that will be experienced when loading an in-box InfiniBand driver co-resident with OPA is a report of "invalid parameters". Syslog will contain reports of "disagrees about version of symbol ib_unregister_mad_agent", etc. This is due to the fact that the updated ib_mad module has a different version of its ABI than what is expected by the in-box drivers. |
10 | |||
11 | 6 | Brian Smith | It is important to note that the InfiniBand SM must not be running on a machine that has OPA installed. The SM must be executed on an IB-only node. Furthermore, the OPA and IB fabrics are orthogonal to each other. There is no direct communication between the two types of fabric. |
12 | 5 | Brian Smith | |
13 | 9 | Brian Smith | ## Building IB Drivers Against OPA-enabled Headers |
14 | 8 | Brian Smith | |
15 | 14 | Brian Smith | In order to use the in-box InfiniBand drivers with the jumbo-MAD-enabled ib_mad module, they must be recompiled against the updated headers. This process is outlined below. The build system must have build-essentials installed. The build should be executed as a non-root user. |
16 | 1 | Brian Smith | |
17 | This process assumes that the mlx4 driver is to be included with kmod-ifs-kernel-updates. It should apply to other InfiniBand drivers. |
||
18 | |||
19 | ``` |
||
20 | # mkdir ifsbuild |
||
21 | # cd ifsbuild |
||
22 | # dget http://security.debian.org/debian-security/pool/updates/main/l/linux/linux_3.16.43-2+deb8u3.dsc |
||
23 | # cp ../IntelOPA-IFS.DEBIAN8-x86_64.10.4.2.0.7-2/packages/ifs-kernel-updates*{orig,debian}* . |
||
24 | # tar xf ifs-kernel-updates_3.16.0-616.orig.tar.gz |
||
25 | # cd ifs-kernel-updates-3.10.0_327.el7.x86_64 |
||
26 | 3 | Brian Smith | # tar xf ../ifs-kernel-updates_3.16.0-616-1ifs.debian.tar.xz |
27 | 1 | Brian Smith | # QUILT_PATCHES=“debian/patches" quilt push -a |
28 | # cp -r ../linux-3.16.43/drivers/infiniband/hw/mlx4/ . |
||
29 | # cd mlx4 |
||
30 | # mv Makefile Makefile.org |
||
31 | 4 | Brian Smith | # cd .. |
32 | 1 | Brian Smith | ``` |
33 | |||
34 | 2 | Brian Smith | ```mlx4/Makefile``` must now be created. The source below will work for the current version of the mlx4 driver (Jessie 8.9). ```Makefile.org``` should be examined to determine the correct directives for ```obj-$(CONFIG_MLX4_INFINIBAND)``` and ```mlx4_ib-y```. |
35 | 1 | Brian Smith | |
36 | ``` |
||
37 | # |
||
38 | # mlx4 module |
||
39 | # |
||
40 | # |
||
41 | # Called from the kernel module build system. |
||
42 | # |
||
43 | ifneq ($(KERNELRELEASE),) |
||
44 | #kbuild part of makefile |
||
45 | |||
46 | ccflags-y := -I$(src)/../include |
||
47 | |||
48 | obj-$(CONFIG_MLX4_INFINIBAND) += mlx4_ib.o |
||
49 | |||
50 | mlx4_ib-y := ah.o cq.o doorbell.o mad.o main.o mr.o qp.o srq.o mcg.o cm.o alias_GUID.o sysfs.o |
||
51 | |||
52 | else |
||
53 | #normal makefile |
||
54 | KDIR ?= /lib/modules/`uname -r`/build |
||
55 | |||
56 | default: |
||
57 | $(MAKE) -C $(KDIR) M=$$PWD NOSTDINC_FLAGS=-I$$PWD |
||
58 | |||
59 | clean: |
||
60 | $(MAKE) -C $(KDIR) M=$$PWD clean |
||
61 | |||
62 | install: |
||
63 | $(MAKE) INSTALL_MOD_DIR=updates -C $(KDIR) M=$$PWD modules_install |
||
64 | |||
65 | endif |
||
66 | ``` |
||
67 | |||
68 | Now, the top-level Makefile should be edited to include mlx4 in the build. Add mlx4/ to the end of the obj-y directive. The trailing slash is necessary. |
||
69 | |||
70 | ``` |
||
71 | obj-y := rdmavt/ ib_uverbs/ hfi1/ ib_mad/ opa_compat/ \ |
||
72 | ib_umad/ ib_sa/ ib_cm/ ib_ucm/ rdma_cm/ rdma_ucm/ ib_srpt/ ib_ipoib/ mlx4/ |
||
73 | ``` |
||
74 | |||
75 | 10 | Brian Smith | It may be desirable to update the package version in ```debian/changelog``` to indicate that the package is different than what was shipped with IFS. An entry may be made at the beginning of the file: |
76 | |||
77 | ```` |
||
78 | ifs-kernel-updates (3.16.0-616-1ifs-mlx4) UNRELEASED; urgency=medium |
||
79 | |||
80 | 11 | Brian Smith | * Add mlx4 module |
81 | 10 | Brian Smith | |
82 | -- Your Name <youremail> Tue, 11 Jul 2017 11:38:22 -0500 |
||
83 | ``` |
||
84 | |||
85 | 7 | Brian Smith | The package may now be built via the command: |
86 | 1 | Brian Smith | |
87 | ``` |
||
88 | # debuild -us -uc -b |
||
89 | ``` |
||
90 | 8 | Brian Smith | |
91 | The ```-b``` flag is necessary because the source does not include a patch to account for the mlx4 additions. If you wish to build a source package and exclude the ```-b``` flag, create a patch with quilt and add Makefile and mlx4/* to the patch. Further explanation of this process is beyond the scope of this document. See https://wiki.debian.org/UsingQuilt for more information. |
||
92 | 1 | Brian Smith | |
93 | The build should complete without errors, although lintian may complain about a few things. The package may now be installed with: |
||
94 | |||
95 | ``` |
||
96 | # sudo dpkg -i ../kmod-ifs-kernel-updates_3.16.0-616-1ifs_amd64.deb |
||
97 | # depmod -a |
||
98 | ``` |
||
99 | |||
100 | The system should now have a recompiled driver installed to ```/lib/modules/3.16.0-4-amd64/updates/ifs-kernel-updates/mlx4/mlx4_ib.ko``` and ```modprobe mlx4_ib``` should execute without errors. ```ibstat``` should include information for the mlx4_0 device. |
||
101 | |||
102 | The system will also need the libmlx4 package installed in order to provide user-space ibverbs access. User-space packages should not require recompilation. |
||
103 | 13 | Brian Smith | |
104 | |||
105 | Brian T. Smith |
||
106 | Senior Technical Staff |
||
107 | System Fabric Works, Inc. |
||
108 | bsmith@systemfabricworks.com |