Pcie passthrough Proxmox
Passing a pcie card through to a VM might seem like a daunting task but it is in fact quite simple. I have three cards passed through to a Windows VM, two sata cards and one graphics card. I will use this machine as an example.
First step is to check for iommu support.
<Pre>
dmesg | grep IOMMU
</Pre>
you should receive a result similar to:
<Pre>
[ 0.000000] DMAR: IOMMU enabled
[ 21.572201] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[ 21.572202] AMD IOMMUv2 functionality not available on this system
</Pre>
Now we add the vfio modules and enable iommu in GRUB
<Pre>
nano /etc/modules
</Pre>
Add these lines
<Pre>
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel
</Pre>
<Pre>
nano /etc/default/grub
</Pre>
Find the line "GRUB_CMDLINE_LINUX_DEFAULT="
for AMD add:
<Pre>
amd_iommu=on
</Pre>
for Intel add:
<Pre>
intel_iommu=on
</Pre>
Now run this script:
<Pre>
#!/bin/sh
if [ $(dmesg | grep ecap | wc -l) -eq 0 ]; then
echo "No interrupt remapping support found"
exit 1
fi
for i in $(dmesg | grep ecap | awk '{print $NF}'); do
if [ $(( (0x$i & 0xf) >> 3 )) -ne 1 ]; then
echo "Interrupt remapping not supported"
exit 1
fi
done
</Pre>
This will check for intrrrupt remapping support. If not supported you also need to add:
<Pre>
vfio_iommu_type1.allow_unsafe_interrupts=1
</Pre>
Your host is now ready for basic passthrough. Easy huh?
Now we get to actually add the cards to the VM.
Run the command:
<Pre>
lspci
</Pre>
My output looks like
<div style="height:30em; overflow:auto; border: 2px solid #088">
<Pre>
00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13)
00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13)
00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 13)
00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 13)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 13)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 RAID bus controller: Intel Corporation SATA Controller [RAID mode]
01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5764M Gigabit Ethernet PCIe (rev 10)
03:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
0f:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV620 LE [Radeon HD 3450]
1c:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller (rev 11)
28:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]
28:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]
37:04.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 10)
37:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
3f:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05)
3f:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05)
3f:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05)
3f:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05)
3f:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05)
3f:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05)
3f:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05)
3f:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05)
3f:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers (rev 05)
3f:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05)
3f:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05)
3f:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05)
3f:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05)
3f:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05)
3f:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05)
3f:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05)
3f:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05)
3f:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05)
3f:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05)
3f:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05)
3f:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05)
3f:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)
</Pre>
</div>
The hex code at the beginning of each line is the address of the pci device. keep in mind that most of these are part of the chipset of the motherboard and not physical cards. I want to add both of the SATA controllers to the VM with the vmid of 230.
<Pre>
nano /etc/pve/qemu-server/230.conf
</Pre>
Add the lines:
<Pre>
cpu: host
machine: q35
hostpci0: 03:00.0,pcie=1
hostpci1: 1c:00.0,pcie=1
</Pre>
Now because my GPU that i want to passthrough has hdmi out it reads as both a video card and an audio card. we can either pass them individually or together. no wrong answer here.
Either:
<Pre>
hostpci2: 28:00.0,pci=1,x-vga=1
hostpci3: 28:00.1,pci=1
</Pre>
Or:
<Pre>
hostpci2: 28:00,pcie=1,x-vga=1
</Pre>
Now reboot and install drivers. you're good to go.