ixgbe{vf} - Physical Function gets IRQ when VF checks link state

Bug #1836760 reported by Guilherme G. Piccoli
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
High
Guilherme G. Piccoli
Trusty
Fix Released
High
Guilherme G. Piccoli
Xenial
Fix Released
High
Guilherme G. Piccoli
Bionic
Fix Released
High
Guilherme G. Piccoli
Cosmic
Won't Fix
High
Guilherme G. Piccoli
Disco
Fix Released
High
Guilherme G. Piccoli
Eoan
Fix Released
High
Guilherme G. Piccoli
Focal
Fix Released
High
Guilherme G. Piccoli

Bug Description

[Impact]

* Intel NICs that are SR-IOV capable and are managed by ixgbe driver presents a potentially harmful behavior when the ixgbevf-managed VFs (Virtual Functions) perform an ethtool link check. The ixgbevf driver issues a mailbox command in the ethtool link state handler, which induces one IRQ in the PF (Physical Function) per link check.

* This was reported as a sort of "denial-of-service" from a guest; due to some link check loop running inside a guest with PCI-PT of a ixgbevf-managed VF, the host received a huge amount of IRQs causing soft-lockups.

* The patch proposed in this SRU request fix this behavior by relying in the saved link state (obtained in the ixgbevf's watchdog routine) instead of issuing a mailbox command to the PF in every link state check request. The commit is available on Linus tree: 1e1b0c658d9b ("ixgbevf: Use cached link state instead of re-reading the value for ethtool")
http://git.kernel.org/linus/1e1b0c658d9b

[Test case]

Reproducing the behavior is pretty simple; having a machine with an Intel NIC managed by ixgbe, proceed with the following steps:

a) Create one or more VFs
(echo 1 > /sys/class/net/<PF iface>/device/sriov_numvfs)

b) In a different terminal, monitor the non-TxRx PF IRQs:
(watch -n1 "cat /proc/interrupts | grep <PF iface> | grep -v Tx")

c) Run "ethtool <VF iface>" in a loop

Without the hereby proposed patch, the PF IRQs will increase.

[Regression potential]

The patch scope is restricted to ixgbevf ethtool link-check procedure, and was developed by the vendor itself. Being a self-contained patch affecting only this driver's ethtool handler, the worst potential regression would be a wrong link state report.

summary: - ixgbe - Physical Function gets IRQ when VF checks link state
+ ixgbe{vf} - Physical Function gets IRQ when VF checks link state
Changed in linux (Ubuntu Disco):
importance: Undecided → High
Changed in linux (Ubuntu Cosmic):
importance: Undecided → High
Changed in linux (Ubuntu Bionic):
importance: Undecided → High
Changed in linux (Ubuntu Xenial):
importance: Undecided → High
Changed in linux (Ubuntu Disco):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
Changed in linux (Ubuntu Cosmic):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
Changed in linux (Ubuntu Bionic):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
Changed in linux (Ubuntu Xenial):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
Changed in linux (Ubuntu Disco):
status: New → Confirmed
Changed in linux (Ubuntu Cosmic):
status: New → Confirmed
Changed in linux (Ubuntu Bionic):
status: New → Confirmed
Changed in linux (Ubuntu Xenial):
status: New → Confirmed
description: updated
Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

SRU request was sent to kernel-team mailing list: https://lists.ubuntu.com/archives/kernel-team/2019-July/102282.html

Cheers,

Guilherme

Changed in linux (Ubuntu Xenial):
status: Confirmed → In Progress
Changed in linux (Ubuntu Bionic):
status: Confirmed → In Progress
Changed in linux (Ubuntu Cosmic):
status: Confirmed → Won't Fix
Changed in linux (Ubuntu Eoan):
status: Confirmed → In Progress
Changed in linux (Ubuntu Disco):
status: Confirmed → In Progress
Seth Forshee (sforshee)
Changed in linux (Ubuntu Eoan):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Disco):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Brad Figg (brad-figg)
tags: added: cscc
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-disco' to 'verification-done-disco'. If the problem still exists, change the tag 'verification-needed-disco' to 'verification-failed-disco'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-disco
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

I've validated the -proposed kernels for Xenial (4.4.0-158), Bionic (4.15.0-56) and Disco (5.0.0-22), using the test case mentioned in the description. All working fine, the issue is gone.
Also, the patch was released upstream in the 5.3.x series, so I'll mark ff-series as Released.

Cheers,

Guilherme

tags: added: verification-done-bionic verification-done-disco verification-done-xenial
removed: verification-needed-bionic verification-needed-disco verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (37.9 KiB)

This bug was fixed in the package linux - 5.2.0-10.11

---------------
linux (5.2.0-10.11) eoan; urgency=medium

  * eoan/linux: 5.2.0-10.11 -proposed tracker (LP: #1838113)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

  * Eoan update: v5.2.4 upstream stable release (LP: #1838428)
    - bnx2x: Prevent load reordering in tx completion processing
    - caif-hsi: fix possible deadlock in cfhsi_exit_module()
    - hv_netvsc: Fix extra rcu_read_unlock in netvsc_recv_callback()
    - igmp: fix memory leak in igmpv3_del_delrec()
    - ipv4: don't set IPv6 only flags to IPv4 addresses
    - ipv6: rt6_check should return NULL if 'from' is NULL
    - ipv6: Unlink sibling route in case of failure
    - net: bcmgenet: use promisc for unsupported filters
    - net: dsa: mv88e6xxx: wait after reset deactivation
    - net: make skb_dst_force return true when dst is refcounted
    - net: neigh: fix multiple neigh timer scheduling
    - net: openvswitch: fix csum updates for MPLS actions
    - net: phy: sfp: hwmon: Fix scaling of RX power
    - net_sched: unset TCQ_F_CAN_BYPASS when adding filters
    - net: stmmac: Re-work the queue selection for TSO packets
    - net/tls: make sure offload also gets the keys wiped
    - nfc: fix potential illegal memory access
    - r8169: fix issue with confused RX unit after PHY power-down on RTL8411b
    - rxrpc: Fix send on a connected, but unbound socket
    - sctp: fix error handling on stream scheduler initialization
    - sctp: not bind the socket in sctp_connect
    - sky2: Disable MSI on ASUS P6T
    - tcp: be more careful in tcp_fragment()
    - tcp: fix tcp_set_congestion_control() use from bpf hook
    - tcp: Reset bytes_acked and bytes_received when disconnecting
    - vrf: make sure skb->data contains ip header to make routing
    - net/mlx5e: IPoIB, Add error path in mlx5_rdma_setup_rn
    - net: bridge: mcast: fix stale nsrcs pointer in igmp3/mld2 report handling
    - net: bridge: mcast: fix stale ipv6 hdr pointer when handling v6 query
    - net: bridge: don't cache ether dest pointer on input
    - net: bridge: stp: don't cache eth dest pointer before skb pull
    - macsec: fix use-after-free of skb during RX
    - macsec: fix checksumming after decryption
    - netrom: fix a memory leak in nr_rx_frame()
    - netrom: hold sock when setting skb->destructor
    - selftests: txring_overwrite: fix incorrect test of mmap() return value
    - net/tls: fix poll ignoring partially copied records
    - net/tls: reject offload of TLS 1.3
    - net/mlx5e: Fix port tunnel GRE entropy control
    - net/mlx5e: Rx, Fix checksum calculation for new hardware
    - net/mlx5e: Fix return value from timeout recover function
    - net/mlx5e: Fix error flow in tx reporter diagnose
    - bnxt_en: Fix VNIC accounting when enabling aRFS on 57500 chips.
    - mlxsw: spectrum_dcb: Configure DSCP map as the last rule is removed
    - net/mlx5: E-Switch, Fix default encap mode
    - mlxsw: spectrum: Do not process learned records with a dummy FID
    - dma-buf: balance refcount inbalance
    - dma-buf: Discard old fence_excl on retrying get_fences_rcu for realloc
    - Revert "gpio/spi: Fix spi-gpio...

Changed in linux (Ubuntu Eoan):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.0.0-25.26

---------------
linux (5.0.0-25.26) disco; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 01 Aug 2019 12:04:35 +0200

Changed in linux (Ubuntu Disco):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (171.3 KiB)

This bug was fixed in the package linux - 4.15.0-58.64

---------------
linux (4.15.0-58.64) bionic; urgency=medium

  * unable to handle kernel NULL pointer dereference at 000000000000002c (IP:
    iget5_locked+0x9e/0x1f0) (LP: #1838982)
    - Revert "ovl: set I_CREATING on inode being created"
    - Revert "new primitive: discard_new_inode()"

linux (4.15.0-57.63) bionic; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

  * Packaging resync (LP: #1786013)
    - update dkms package versions

linux (4.15.0-56.62) bionic; urgency=medium

  * bionic/linux: 4.15.0-56.62 -proposed tracker (LP: #1837626)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update helper scripts

  * CVE-2019-2101
    - media: uvcvideo: Fix 'type' check leading to overflow

  * hibmc-drm Causes Unreadable Display for Huawei amd64 Servers (LP: #1762940)
    - [Config] Set CONFIG_DRM_HISI_HIBMC to arm64 only
    - SAUCE: Make CONFIG_DRM_HISI_HIBMC depend on ARM64

  * Bionic: support for Solarflare X2542 network adapter (sfc driver)
    (LP: #1836635)
    - sfc: make mem_bar a function rather than a constant
    - sfc: support VI strides other than 8k
    - sfc: add Medford2 (SFC9250) PCI Device IDs
    - sfc: improve PTP error reporting
    - sfc: update EF10 register definitions
    - sfc: populate the timer reload field
    - sfc: update MCDI protocol headers
    - sfc: support variable number of MAC stats
    - sfc: expose FEC stats on Medford2
    - sfc: expose CTPIO stats on NICs that support them
    - sfc: basic MCDI mapping of 25/50/100G link speeds
    - sfc: support the ethtool ksettings API properly so that 25/50/100G works
    - sfc: add bits for 25/50/100G supported/advertised speeds
    - sfc: remove tx and MCDI handling from NAPI budget consideration
    - sfc: handle TX timestamps in the normal data path
    - sfc: add function to determine which TX timestamping method to use
    - sfc: use main datapath for HW timestamps if available
    - sfc: only enable TX timestamping if the adapter is licensed for it
    - sfc: MAC TX timestamp handling on the 8000 series
    - sfc: on 8000 series use TX queues for TX timestamps
    - sfc: only advertise TX timestamping if we have the license for it
    - sfc: simplify RX datapath timestamping
    - sfc: support separate PTP and general timestamping
    - sfc: support second + quarter ns time format for receive datapath
    - sfc: support Medford2 frequency adjustment format
    - sfc: add suffix to large constant in ptp
    - sfc: mark some unexported symbols as static
    - sfc: update MCDI protocol headers
    - sfc: support FEC configuration through ethtool
    - sfc: remove ctpio_dmabuf_start from stats
    - sfc: stop the TX queue before pushing new buffers

  * [18.04 FEAT] zKVM: Add hardwar...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (12.0 KiB)

This bug was fixed in the package linux - 4.4.0-159.187

---------------
linux (4.4.0-159.187) xenial; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

linux (4.4.0-158.186) xenial; urgency=medium

  * xenial/linux: 4.4.0-158.186 -proposed tracker (LP: #1837609)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update helper scripts

  * ixgbe{vf} - Physical Function gets IRQ when VF checks link state
    (LP: #1836760)
    - ixgbevf: Use cached link state instead of re-reading the value for ethtool

  * CVE-2018-5383
    - crypto: kpp - Key-agreement Protocol Primitives API (KPP)
    - crypto: dh - Add DH software implementation
    - crypto: ecdh - Add ECDH software support
    - crypto: ecdh - make ecdh_shared_secret unique
    - crypto: doc - add KPP documentation
    - crypto: kpp, (ec)dh - fix typos
    - crypto: ecc - remove unused function arguments
    - crypto: ecc - remove unnecessary casts
    - crypto: ecc - rename ecdh_make_pub_key()
    - crypto: ecdh - add privkey generation support
    - crypto: ecc - Fix NULL pointer deref. on no default_rng
    - [Config] CRYPTO_ECDH=m
    - Bluetooth: convert smp and selftest to crypto kpp API
    - crypto: ecdh - add public key verification test

  * Xenial update: 4.4.185 upstream stable release (LP: #1836668)
    - fs/binfmt_flat.c: make load_flat_shared_library() work
    - scsi: vmw_pscsi: Fix use-after-free in pvscsi_queue_lck()
    - tracing: Silence GCC 9 array bounds warning
    - gcc-9: silence 'address-of-packed-member' warning
    - usb: chipidea: udc: workaround for endpoint conflict issue
    - Input: uinput - add compat ioctl number translation for UI_*_FF_UPLOAD
    - apparmor: enforce nullbyte at end of tag string
    - parport: Fix mem leak in parport_register_dev_model
    - parisc: Fix compiler warnings in float emulation code
    - IB/hfi1: Insure freeze_work work_struct is canceled on shutdown
    - MIPS: uprobes: remove set but not used variable 'epc'
    - net: hns: Fix loopback test failed at copper ports
    - sparc: perf: fix updated event period in response to PERF_EVENT_IOC_PERIOD
    - scripts/checkstack.pl: Fix arm64 wrong or unknown architecture
    - scsi: ufs: Check that space was properly alloced in copy_query_response
    - s390/qeth: fix VLAN attribute in bridge_hostnotify udev event
    - hwmon: (pmbus/core) Treat parameters as paged if on multiple pages
    - Btrfs: fix race between readahead and device replace/removal
    - btrfs: start readahead also in seed devices
    - can: flexcan: fix timeout when set small bitrate
    - can: purge socket error queue on sock destruct
    - ARM: imx: cpuidle-imx6sx: Restrict the SW2ISO increase to i.MX6SX
    - Bluetooth: Align minimum encryption key size for LE and BR/EDR connections
    - Bluet...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Stefan Bader (smb)
Changed in linux (Ubuntu Trusty):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
importance: Undecided → High
status: New → In Progress
Changed in linux (Ubuntu Trusty):
status: In Progress → Fix Committed
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (9.1 KiB)

Verification successful on Ubuntu 14.04 (ESM):

No interrupts are generated on the PF by the link status check of the VF.
Verified with both Link down and Link up.

# uname -rv
3.13.0-174-generic #225-Ubuntu SMP Wed Oct 2 22:08:03 UTC 2019

# ls -ld /sys/bus/pci/drivers/ixgbe/*/net/*
drwxr-xr-x 5 root root 0 Oct 17 20:53 /sys/bus/pci/drivers/ixgbe/0000:04:00.0/net/eth4
drwxr-xr-x 5 root root 0 Oct 17 20:53 /sys/bus/pci/drivers/ixgbe/0000:04:00.1/net/eth5

# echo 1 > /sys/class/net/eth4/device/sriov_numvfs

# dmesg
...
[ 1104.000023] ixgbe 0000:04:00.0 eth4: SR-IOV enabled with 1 VFs
[ 1104.100390] pci 0000:04:10.0: [8086:1515] type 00 class 0x020000
[ 1104.100981] ixgbe 0000:04:00.0: removed PHC on eth4
[ 1104.109683] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.11.3-k
[ 1104.109689] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 1104.109797] ixgbevf 0000:04:10.0: enabling device (0000 -> 0002)
[ 1104.109976] ixgbe 0000:04:00.0 eth4: VF Reset msg received from vf 0
[ 1104.110096] ixgbe 0000:04:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[ 1104.120353] ixgbevf 0000:04:10.0: PF still in reset state. Is the PF interface up?
[ 1104.120359] ixgbevf 0000:04:10.0: Assigning random MAC address
[ 1104.121016] ixgbevf 0000:04:10.0: irq 305 for MSI/MSI-X
[ 1104.121037] ixgbevf 0000:04:10.0: irq 306 for MSI/MSI-X
[ 1104.516319] ixgbe 0000:04:00.0: irq 155 for MSI/MSI-X
[ 1104.516335] ixgbe 0000:04:00.0: irq 156 for MSI/MSI-X
[ 1104.516359] ixgbe 0000:04:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 1104.523447] pps pps1: new PPS source ptp4
[ 1104.523452] ixgbe 0000:04:00.0: registered PHC device on eth4
[ 1111.947668] ixgbe 0000:04:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: None

# ls -ld /sys/bus/pci/drivers/ixgbevf/*/net/*
drwxr-xr-x 5 root root 0 Oct 17 21:11 /sys/bus/pci/drivers/ixgbevf/0000:04:10.0/net/enp4s16

Link down:

# ethtool enp4s16 | grep Link
 Link detected: no

# while true; do ethtool enp4s16 >/dev/null; done

On another terminal:

$ while sleep 1; do echo -n "$(date) - "; cat /proc/interrupts | grep eth4 | grep -v Tx | tr -s ' '; done
Thu Oct 17 21:21:08 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:09 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:10 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:11 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:12 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:13 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-e...

Read more...

Changed in linux (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.