Project

General

Profile

KVM PCI Passthrough and Omni-Path » History » Version 13

Brian Smith, 04/07/2018 02:36 AM

1 1 Brian Smith
# KVM PCI Passthrough and OPA
2
3 11 Brian Smith
A KVM guest can use OPA hardware when configured for PCI passthrough. This document is OPA and Debian-centric, but the concepts should apply to other host operating systems and PCI devices.
4 1 Brian Smith
5
## BIOS Settings
6
7
1. Intel VT must be enabled.
8 4 Brian Smith
2. Integrated IO / IntelVT must be enabled.
9 1 Brian Smith
10
## Kernel Command Line
11
12 9 Brian Smith
Add this to the host's kernel command line and reboot the host:
13
14 10 Brian Smith
```
15
intel_iommu=on iommu=pt
16
```
17 1 Brian Smith
18 5 Brian Smith
When configured properly, ```/sys/kernel/iommu_groups/``` will contain many subdirectories. If that path is empty, IOMMU is not working.
19
20 1 Brian Smith
## Install KVM
21
22
```
23 13 Brian Smith
$ sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system virtinst libosinfo-bin virsh 
24 1 Brian Smith
$ sudo adduser YOU libvirt
25
$ sudo adduser YOU libvirt-qemu
26
$ sudo adduser YOU kvm
27
```
28
29
## Disable hfi1 on host
30
31
The hfi1 driver must not be loaded on the host machine, in order to use PCI passthrough. In /etc/modprobe.d/hfi1.conf:
32
33
```
34
blacklist hfi1
35
```
36
37
Also, there is no reason to have IFS installed on the host. The host machine should have no OPA functionality enabled.
38
39
## Configure PCI Passthrough
40
41
The hfi1 device must be setup for PCI passthrough.  Find the device's port in the output of lspci:
42
43
```
44
$ lspci | grep Omni | cut -f1 '-d '
45
```
46
47
For the scripts below, prepend the port with 0000:, like "0000:80:02.0".
48
49
50
Use the following script, replace PCI_PORT with the port of the hfi1:
51
52
```
53
#!/bin/bash
54
55
PCI_PORT=0000:80:02.0
56
DEV_VENDOR=8086
57
DEV_MODEL=24f0
58
59
rmmod vfio
60
rmmod vfio_pci
61
echo "$PCI_PORT" > /sys/bus/pci/devices/$PCI_PORT/driver/unbind
62
modprobe vfio
63
modprobe vfio_pci
64
echo $DEV_VENDOR $DEV_MODEL > /sys/bus/pci/drivers/vfio-pci/new_id
65
```
66
67 2 Brian Smith
## Configure Default Network for DNS Forwarding
68
69
```
70
$ sudo virsh net-edit default
71
```
72
73
Add this tag:
74
75
```
76
  <domain name='sfw.int' localOnly='no'/>
77
```
78
79 1 Brian Smith
## Create Guest
80
81 3 Brian Smith
While it is possible to manage guests for an unprivileged user, they get a non-functional network setup in the default config. TBD to figure this out. 
82
83
**Use virsh as root.**
84 1 Brian Smith
85
```
86
$ systemctl start libvirtd
87 8 Brian Smith
$ virt-install --virt-type kvm --name GUEST_NAME \
88 1 Brian Smith
    --vcpus=4 --virt-type kvm --cdrom $HOME/kvm-guest/debian-8.7.0-amd64-DVD-1.iso \
89
    -v --os-variant debian8 \
90 8 Brian Smith
    --disk path=PATH_TO_CREATE_DISK,size=16 --memory 4096 --graphics vnc
91 1 Brian Smith
```
92
93
Connect a VNC client to a tunneled connection to the host.
94
95
From the workstation:
96
97
```
98
$ ssh -L5910:host:5900 YOU@host
99
```
100
101
Now connect a VNC client to localhost:5910 and complete the install.
102
103
## Import Existing Disk to New Guest
104
105
To import an existing guest disk image, use the following command:
106
107
```
108 7 Brian Smith
$ sudo virt-install --virt-type kvm --name GUEST_NAME \
109 1 Brian Smith
    --vcpus=4 --virt-type kvm --import \
110
    -v --os-variant debian8 \
111
    --disk PATH_TO_DISK_IMAGE,device=disk,bus=virtio --memory 4096 --graphics vnc
112
```
113 2 Brian Smith
114
## Connect to Guest, Configure DNS
115
116
TBD: figure out bridged network
117
118
The default network for KVM is 192.168.122.0/24 and the guest should be assigned a DHCP address when it boots. Use the VNC connection to execute ```$ ip addr``. ssh should be able to connect to the guest from the host.
119
120
Unfortunately, dnsmasq doesn't appear to set the search domain properly. For Debian, configure a search domain in the guest's ```/etc/network/interfaces```.
121
122
```
123
allow-hotplug eth0
124
iface eth0 inet dhcp
125
    dns-search sfw.int
126
```
127
128 1 Brian Smith
## Configure Guest for PCI Passthrough
129
130
Shutdown the guest if it is running.
131
132
```
133
$ virsh shutdown GUEST
134
```
135
136
Look for the PCI device in virsh. Look for a pci device that matches the port found via lspci.
137
138
```
139
$ virsh nodedev-list --tree 
140
```
141
142
Detach the device. Use the child device of the one that matches the device you found via lspci.
143
144
```
145
$ virsh nodedev-detach pci_0000_81_00_0
146
```
147
148
Dump the device info.
149
150
```
151
$ virsh nodedev-dumpxml pci_0000_81_00_0
152
```
153
154
Convert bus, slot and function to hex. ```$ printf %x VALUE``` can be used for the hex-challenged.
155
156
Edit the guest and add a hostdev section:
157
158
```
159
<hostdev mode='subsystem' type='pci' managed='yes'>
160
  <source>
161
      <address domain='0x0000' bus='0x81' slot='0x0' function='0x0'/>
162
  </source>
163
</hostdev>
164
```
165
166
Upon booting the guest, the passthrough device should be present in the guest's lspci output.
167
168 12 Brian Smith
**Note**: the PCI device will have different capabilities in the VM than it has on the physical host. Hopefully, the driver takes this into account. Refer to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4c009af473b2026caaa26107e34d7cc68dad7756 for a patch that fixes one such problem in hfi1. Hope it helps.
169 1 Brian Smith
170
## References
171
172
1. https://wiki.debian.org/KVM
173
2. https://jamielinux.com/docs/libvirt-networking-handbook/nat-based-network.html
174
3. https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
175
4. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
176
5. https://wiki.debian.org/VGAPassthrough
177
178
----
179
180
{{lastupdated_by}} {{lastupdated_at}}
181
182
{{comment_form}}
183
{{comments}}