Project

General

Profile

KVM PCI Passthrough and Omni-Path » History » Version 5

Brian Smith, 04/07/2018 02:01 AM

1 1 Brian Smith
# KVM PCI Passthrough and OPA
2
3
A KVM guest can use OPA hardware when configured for PCI passthrough.
4
5
## BIOS Settings
6
7
1. Intel VT must be enabled.
8 4 Brian Smith
2. Integrated IO / IntelVT must be enabled.
9 1 Brian Smith
10
## Kernel Command Line
11
12
```intel_iommu=on iommu=pt```
13
14 5 Brian Smith
When configured properly, ```/sys/kernel/iommu_groups/``` will contain many subdirectories. If that path is empty, IOMMU is not working.
15
16 1 Brian Smith
## Install KVM
17
18
```
19
$ sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system virtinst libosinfo-bin virt-viewer virsh 
20
$ sudo adduser YOU libvirt
21
$ sudo adduser YOU libvirt-qemu
22
$ sudo adduser YOU kvm
23
```
24
25
## Disable hfi1 on host
26
27
The hfi1 driver must not be loaded on the host machine, in order to use PCI passthrough. In /etc/modprobe.d/hfi1.conf:
28
29
```
30
blacklist hfi1
31
```
32
33
Also, there is no reason to have IFS installed on the host. The host machine should have no OPA functionality enabled.
34
35
## Configure PCI Passthrough
36
37
The hfi1 device must be setup for PCI passthrough.  Find the device's port in the output of lspci:
38
39
```
40
$ lspci | grep Omni | cut -f1 '-d '
41
```
42
43
For the scripts below, prepend the port with 0000:, like "0000:80:02.0".
44
45
46
Use the following script, replace PCI_PORT with the port of the hfi1:
47
48
```
49
#!/bin/bash
50
51
PCI_PORT=0000:80:02.0
52
DEV_VENDOR=8086
53
DEV_MODEL=24f0
54
55
rmmod vfio
56
rmmod vfio_pci
57
echo "$PCI_PORT" > /sys/bus/pci/devices/$PCI_PORT/driver/unbind
58
modprobe vfio
59
modprobe vfio_pci
60
echo $DEV_VENDOR $DEV_MODEL > /sys/bus/pci/drivers/vfio-pci/new_id
61
```
62
63 2 Brian Smith
## Configure Default Network for DNS Forwarding
64
65
```
66
$ sudo virsh net-edit default
67
```
68
69
Add this tag:
70
71
```
72
  <domain name='sfw.int' localOnly='no'/>
73
```
74
75 1 Brian Smith
## Create Guest
76
77 3 Brian Smith
While it is possible to manage guests for an unprivileged user, they get a non-functional network setup in the default config. TBD to figure this out. 
78
79
**Use virsh as root.**
80 1 Brian Smith
81
```
82
$ systemctl start libvirtd
83
$ virt-install --virt-type kvm --name gozer-deb8 \
84
    --vcpus=4 --virt-type kvm --cdrom $HOME/kvm-guest/debian-8.7.0-amd64-DVD-1.iso \
85
    -v --os-variant debian8 \
86
    --disk path=$HOME/kvm-guest/gozer-deb8-d1.disk,size=16 --memory 4096 --graphics vnc
87
```
88
89
Connect a VNC client to a tunneled connection to the host.
90
91
From the workstation:
92
93
```
94
$ ssh -L5910:host:5900 YOU@host
95
```
96
97
Now connect a VNC client to localhost:5910 and complete the install.
98
99
## Import Existing Disk to New Guest
100
101
To import an existing guest disk image, use the following command:
102
103
```
104 2 Brian Smith
$ sudo virt-install --virt-type kvm --name gozer-deb8 \
105 1 Brian Smith
    --vcpus=4 --virt-type kvm --import \
106
    -v --os-variant debian8 \
107
    --disk PATH_TO_DISK_IMAGE,device=disk,bus=virtio --memory 4096 --graphics vnc
108
```
109 2 Brian Smith
110
## Connect to Guest, Configure DNS
111
112
TBD: figure out bridged network
113
114
The default network for KVM is 192.168.122.0/24 and the guest should be assigned a DHCP address when it boots. Use the VNC connection to execute ```$ ip addr``. ssh should be able to connect to the guest from the host.
115
116
Unfortunately, dnsmasq doesn't appear to set the search domain properly. For Debian, configure a search domain in the guest's ```/etc/network/interfaces```.
117
118
```
119
allow-hotplug eth0
120
iface eth0 inet dhcp
121
    dns-search sfw.int
122
```
123
124 1 Brian Smith
125
## Configure Guest for PCI Passthrough
126
127
Shutdown the guest if it is running.
128
129
```
130
$ virsh shutdown GUEST
131
```
132
133
Look for the PCI device in virsh. Look for a pci device that matches the port found via lspci.
134
135
```
136
$ virsh nodedev-list --tree 
137
```
138
139
Detach the device. Use the child device of the one that matches the device you found via lspci.
140
141
```
142
$ virsh nodedev-detach pci_0000_81_00_0
143
```
144
145
Dump the device info.
146
147
```
148
$ virsh nodedev-dumpxml pci_0000_81_00_0
149
```
150
151
Convert bus, slot and function to hex. ```$ printf %x VALUE``` can be used for the hex-challenged.
152
153
Edit the guest and add a hostdev section:
154
155
```
156
<hostdev mode='subsystem' type='pci' managed='yes'>
157
  <source>
158
      <address domain='0x0000' bus='0x81' slot='0x0' function='0x0'/>
159
  </source>
160
</hostdev>
161
```
162
163
Upon booting the guest, the passthrough device should be present in the guest's lspci output.
164
165
166
## References
167
168
1. https://wiki.debian.org/KVM
169
2. https://jamielinux.com/docs/libvirt-networking-handbook/nat-based-network.html
170
3. https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
171
4. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
172
5. https://wiki.debian.org/VGAPassthrough
173
174
----
175
176
{{lastupdated_by}} {{lastupdated_at}}
177
178
{{comment_form}}
179
{{comments}}