IntelOPA - Intel Omni-Path Fabric Suite for Debian

# KVM PCI Passthrough and Omni-Path

A KVM guest can use OPA hardware when configured for PCI passthrough. This document is OPA and Debian-centric, but the concepts should apply to other Linux host operating systems and PCI devices.

## BIOS Settings

1. Intel VT must be enabled.

2. Integrated IO / IntelVT must be enabled.

## Kernel Command Line

Add this to the host's kernel command line and reboot the host:

```

intel_iommu=on iommu=pt

```

When configured properly, ```/sys/kernel/iommu_groups/``` will contain many subdirectories. If that path is empty, IOMMU is not working.

## Install KVM

```

$ sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system virtinst libosinfo-bin 

```

## Disable hfi1 on host

The hfi1 driver must not be loaded on the host machine, in order to use PCI passthrough. In /etc/modprobe.d/hfi1.conf:

```

blacklist hfi1

```

Also, there is no reason to have IFS installed on the host. The host machine should have no OPA functionality enabled.

## Configure PCI Passthrough

The hfi1 device must be setup for PCI passthrough.  Find the device's port in the output of lspci:

```

$ lspci -vnn | grep Omni | cut -f1 '-d '

```

For the scripts below, prepend the port with 0000:, like "0000:80:02.0".

Use the following script, replace PCI_PORT with the port of the hfi1:

```

#!/bin/bash

PCI_PORT=0000:80:02.0

DEV_VENDOR=8086

DEV_MODEL=24f0

rmmod vfio_pci

rmmod vfio

echo "$PCI_PORT" > /sys/bus/pci/devices/$PCI_PORT/driver/unbind

modprobe vfio

modprobe vfio_pci

echo $DEV_VENDOR $DEV_MODEL > /sys/bus/pci/drivers/vfio-pci/new_id

```

## Create Guest

While it is possible to manage guests for an unprivileged user, they get a non-functional network setup in the default config.

**Use virsh as root.**

```

$ systemctl start libvirtd

$ virt-install --name GUEST_NAME \

    --vcpus=4 --virt-type kvm --cdrom $HOME/kvm-guest/debian-8.7.0-amd64-DVD-1.iso \

    -v --os-variant debian8 \

    --disk path=PATH_TO_CREATE_DISK,size=16 --memory 4096 --graphics vnc

```

Connect a VNC client to a tunneled connection to the host.

From the workstation:

```

$ ssh -L5910:localhost:5900 YOU@HOST

```

Now connect a VNC client to localhost:5910 and complete the install.

## Import Existing Disk to New Guest

To import an existing guest disk image, use the following command:

```

$ sudo virt-install --virt-type kvm --name GUEST_NAME \

    --vcpus=4 --import \

    -v --os-variant debian8 \

    --disk PATH_TO_DISK_IMAGE,device=disk,bus=virtio --memory 4096 --graphics vnc

```

## Connect to Guest, Configure DNS

The default network for KVM is 192.168.122.0/24 and the guest should be assigned a DHCP address when it boots. Use the VNC connection to execute ```$ ip addr``. ssh should be able to connect to the guest from the host.

Unfortunately, dnsmasq doesn't appear to set the search domain properly. For Debian, configure a search domain in the guest's ```/etc/network/interfaces```.

```

allow-hotplug eth0

iface eth0 inet dhcp

    dns-search MYDOMAIN

```

## Configure Guest for PCI Passthrough

Shutdown the guest if it is running.

```

$ virsh shutdown GUEST_NAME

```

Look for the PCI device in virsh. Look for a pci device that matches the port found via lspci.

```

$ virsh nodedev-list --tree 

```

Detach the device. Use the child device of the one that matches the device you found via lspci.

```

$ virsh nodedev-detach pci_0000_81_00_0

```

Dump the device info.

```

$ virsh nodedev-dumpxml pci_0000_81_00_0

```

Convert bus, slot and function to hex. The printf utility may be used to do this.

```

$ printf %x VALUE

``` 

Edit the guest and add a hostdev section:

```

$ virsh edit GUEST_NAME

<hostdev mode='subsystem' type='pci' managed='yes'>

  <source>

      <address domain='0x0000' bus='0x81' slot='0x0' function='0x0'/>

  </source>

</hostdev>

```

Boot the guest

```

$ virsh start GUEST_NAME

```

Upon booting the guest, the passthrough device should be present in the guest's lspci output. The passthrough device should be usable by the guest's kernel drivers.

**Note**: the PCI device may have different capabilities in the VM than it has on the physical host. Hopefully, the driver takes this into account. Refer to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4c009af473b2026caaa26107e34d7cc68dad7756 for a patch that fixes one such problem in hfi1. Hope it helps.

## References

1. https://wiki.debian.org/KVM

2. https://jamielinux.com/docs/libvirt-networking-handbook/nat-based-network.html

3. https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

4. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

5. https://wiki.debian.org/VGAPassthrough

----

Brian T. Smith

Senior Technical Staff

System Fabric Works, Inc.

bsmith@systemfabricworks.com