in network

Enable SR-IOV on Proxmox CE

What is SR-IOV?

In network virtualization, a single root input/output virtualization or SR-IOV is a network interface that allows the isolation of the PCI Express resources for manageability and performance reasons

Environment

  • Proxmox
    • pve-manager/4.4-13/7ea56165 (running kernel: 4.4.49-1-pve)
    • Based on Debian 8 (jessie)
  • NIC
    • Intel Corporation Ethernet Controller X710
    • Logical NIC
    • eth{2..5}

Processes

Should enable IOMMU

# Make sure enable SR-IOV function and Intel VT-d in BIOS

# Append configuration
root@pve:~# cat /etc/default/grub |grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet processor.max_cstates=1 idle=poll pcie_aspm=off intel_iommu=on"

# Update GRUB configuration 
root@pve:~# grub2-mkconfig –o /boot/grub2/grub.cfg

# Rebbot for the iommu change to take effect
root@pve:~# reboot

# Check kernel command line in Linux and look for `intel_iommu=on`
root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.49-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on processor.max_cstates=1 idle=poll pcie_aspm=off

# Check dmesg log
root@pve:~# dmesg | grep IOMMU | grep enabled
[    0.000000] DMAR: IOMMU enabled

Create Virtual Functions (VFs)

  • Linux does not create VFs by default. The X710 support upto 32VFs per port. The XL710 support upto 64VFs per port.
# For example(eth2), X710 support 32VFs per physical port
root@pve:~# cat /sys/class/net/eth2/device/sriov_totalvfs
32
  • Enabled with kernel parameter intel_iommu
  • Please Append intel_iommu=on to the GRUB_CMDLINE_LINUX entry in /etc/default/grub if not exist
root@pve:~# echo "4" > /sys/class/net/eth2/device/sriov_numvfs
root@pve:~# cat /sys/class/net/eth2/device/sriov_numvfs
4

root@pve:~# lspci | grep -i 'Virtual Function'
01:02.0 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02)
01:02.1 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02)
01:02.2 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02)
01:02.3 Ethernet controller: Intel Corporation XL710/X710 Virtual Function (rev 02)
  • Create 4 VFs on eth2 (You can choose any interface name which is in X710 adapter)
root@pve:~# ip link set dev eth2 up
root@pve:~# ip link set dev eth2 vf 0 mac aa:bb:cc:dd:ee:00
root@pve:~# ip link set dev eth2 vf 1 mac aa:bb:cc:dd:ee:01
root@pve:~# ip link set dev eth2 vf 2 mac aa:bb:cc:dd:ee:02
root@pve:~# ip link set dev eth2 vf 3 mac aa:bb:cc:dd:ee:03
  • Bring up link on VFs
    Tip: If a VFs MAC address is not assigned in the host, then the VF driver will use a random MAC address. This random MAC address may change each time the VF driver is reloaded. You can assign a static MAC address in the host machine. This static MAC address will survive a VF driver reload.
root@pve:~# rmmod i40evf
root@pve:~# modprobe i40evf

# 10G Intel NIC module name: ixgbevf
  • Reload vf module
root@pve:~# ip link show eth2
3: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq portid 8cea1b30da53 state DOWN mode DEFAULT group default qlen 1000
    link/ether 8c:ea:1b:30:da:53 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC aa:bb:cc:dd:ee:00, spoof checking on, link-state auto
    vf 1 MAC aa:bb:cc:dd:ee:01, spoof checking on, link-state auto
    vf 2 MAC aa:bb:cc:dd:ee:02, spoof checking on, link-state auto
    vf 3 MAC aa:bb:cc:dd:ee:03, spoof checking on, link-state auto
  • Check eth2 Status
# iproute2
root@pve:~# ip link show eth6
root@pve:~# ip link show eth7
root@pve:~# ip link show eth8
root@pve:~# ip link show eth9

# ethtool
root@pve:~# ethtool -i eth6
driver: i40evf
version: 1.4.15-k
firmware-version: N/A
bus-info: 0000:01:02.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
  • Check interface of VFs

Screenshot from Proxmox

References