arm64: Unfair rwlock can stall the system

Bug #1732238 reported by dann frazier
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
dann frazier
Artful
Fix Released
Undecided
dann frazier

Bug Description

[Impact]
There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability.

[Test Case]
$ stress-ng --kill 0 -t 300 -v

You'll see the console fill with messages like:

[ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 2534.428606] 192-...: (1 ticks this GP) idle=b6e/140000000000000/0 softirq=578/578 fqs=6770
[ 2534.437029] (detected by 0, t=15005 jiffies, g=1479, c=1478, q=473)
[ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 2714.629181] 192-...: (1 ticks this GP) idle=b6e/140000000000000/0 softirq=578/578 fqs=12819
[ 2714.637692] (detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736)
[ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464]
[ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936]

[Regression Risk]
The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1732238

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
dann frazier (dannf)
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu Artful):
status: New → Confirmed
Changed in linux (Ubuntu):
assignee: nobody → dann frazier (dannf)
Changed in linux (Ubuntu Artful):
assignee: nobody → dann frazier (dannf)
Revision history for this message
dann frazier (dannf) wrote :

This has been resolved with the following commits upstream:

commit d133166146333e1f13fc81c0e6c43c8d99290a8a
Author: Will Deacon <email address hidden>

    locking/qrwlock: Prevent slowpath writers getting held up by fastpath

commit 087133ac90763cd339b6b67f2998f87dcc136c52
Author: Will Deacon <email address hidden>

    locking/qrwlock, arm64: Move rwlock implementation over to qrwlocks

commit b519b56e378ee82caf9b079b04f5db87dedc3251
Author: Will Deacon <email address hidden>

    locking/qrwlock: Use atomic_cond_read_acquire() when spinning in qrwlock

commit 4df714be4dcf40bfb0d4af0f851a6e1977afa02e
Author: Will Deacon <email address hidden>

    locking/atomic: Add atomic_cond_read_acquire()

commit e0d02285f16e8d5810f3d5d5e8a5886ca0015d3b
Author: Will Deacon <email address hidden>

    locking/qrwlock: Use 'struct qrwlock' instead of 'struct __qrwlock

dann frazier (dannf)
description: updated
Revision history for this message
dann frazier (dannf) wrote :
Download full text (6.6 KiB)

Example showing driver timeouts:

ubuntu@boomer:~$ stress-ng --kill 0 -t 300 -v
stress-ng: debug: [3344] 224 processors online, 224 processors configured
stress-ng: info: [3344] dispatching hogs: 224 kill
stress-ng: debug: [3344] /sys/devices/system/cpu/cpu0/cache does not exist
stress-ng: info: [3344] cache allocate: using built-in defaults as unable to determine cache details
stress-ng: info: [3344] cache allocate: default cache size: 2048K
stress-ng: debug: [3344] starting stressors
stress-ng: debug: [3345] stress-ng-kill: started [3345] (instance 0)
stress-ng: debug: [3346] stress-ng-kill: started [3346] (instance 1)
stress-ng: debug: [3347] stress-ng-kill: started [3347] (instance 2)
stress-ng: debug: [3348] stress-ng-kill: started [3348] (instance 3)
stress-ng: debug: [3349] stress-ng-kill: started [3349] (instance 4)
[ 1447.474535] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1447.480020] 27-...: (66 GPs behind) idle=1ba/140000000000000/0 softirq=3878/3878 fqs=7264
[ 1447.488363] 136-...: (93 GPs behind) idle=972/140000000000000/0 softirq=2760/2760 fqs=7265
[ 1447.496788] (detected by 161, t=15007 jiffies, g=1128, c=1127, q=790)
[ 1451.646152] xhci_hcd 0000:01:04.1: xHCI host controller not responding, assume dead
[ 1451.653819] xhci_hcd 0000:01:04.1: HC died; cleaning up
[ 1451.653829] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653832] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653833] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653834] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653835] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653837] usb 3-1-port1: Cannot enable. Maybe the USB cable is bad?
[ 1451.653839] usb 3-1-port1: cannot disable (err = -22)
[ 1451.653848] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653851] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653852] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653854] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653855] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653856] usb 3-1-port2: Cannot enable. Maybe the USB cable is bad?
[ 1451.653858] usb 3-1-port2: cannot disable (err = -22)
[ 1451.653860] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653861] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653862] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653863] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653864] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653865] usb 3-1-port1: Cannot enable. Maybe the USB cable is bad?
[ 1451.653866] usb 3-1-port1: cannot disable (err = -22)
[ 1451.653868] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653870] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653871] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653872] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653873] usb 3-1-port2: cannot reset (err = -22)
[ 1451.653873] usb 3-1-port2: Cannot enable. Maybe the USB cable is bad?
[ 1451.653875] usb 3-1-port2: cannot disable (err = -22)
[ 1451.653876] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653878] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653879] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653880] usb 3-1-port1: cannot reset (err = -22)
[ 1451.653881] usb 3-1-port1: cannot reset (err ...

Read more...

dann frazier (dannf)
description: updated
Seth Forshee (sforshee)
Changed in linux (Ubuntu):
status: Confirmed → Fix Committed
Changed in linux (Ubuntu Artful):
status: Confirmed → Fix Committed
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-artful
Revision history for this message
dann frazier (dannf) wrote :

Artful verification: I was able to successfully run the above stress-ng commmand w/o any errors on the console.

tags: added: verification-done-artful
removed: verification-needed-artful
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (20.1 KiB)

This bug was fixed in the package linux - 4.13.0-36.40

---------------
linux (4.13.0-36.40) artful; urgency=medium

  * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010)

  * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set

linux (4.13.0-35.39) artful; urgency=medium

  * linux: 4.13.0-35.39 -proposed tracker (LP: #1748743)

  * CVE-2017-5715 (Spectre v2 Intel)
    - Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present"
    - SAUCE: turn off IBRS when full retpoline is present
    - [Packaging] retpoline files must be sorted
    - [Packaging] pull in retpoline files

linux (4.13.0-34.37) artful; urgency=medium

  * linux: 4.13.0-34.37 -proposed tracker (LP: #1748475)

  * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053)
    - libata: apply MAX_SEC_1024 to all LITEON EP1 series devices

  * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb)
    (LP: #1747090)
    - KVM: s390: wire up bpb feature

  * artful 4.13 i386 kernels crash after memory hotplug remove (LP: #1747069)
    - Revert "mm, memory_hotplug: do not associate hotadded memory to zones until
      online"

  * CVE-2017-5715 (Spectre v2 Intel)
    - x86/feature: Enable the x86 feature to control Speculation
    - x86/feature: Report presence of IBPB and IBRS control
    - x86/enter: MACROS to set/clear IBRS and set IBPB
    - x86/enter: Use IBRS on syscall and interrupts
    - x86/idle: Disable IBRS entering idle and enable it on wakeup
    - x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup
    - x86/mm: Set IBPB upon context switch
    - x86/mm: Only set IBPB when the new thread cannot ptrace current thread
    - x86/entry: Stuff RSB for entry to kernel for non-SMEP platform
    - x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm
    - x86/kvm: Set IBPB when switching VM
    - x86/kvm: Toggle IBRS on VM entry and exit
    - x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature
    - x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control
    - x86/cpu/AMD: Add speculative control support for AMD
    - x86/microcode: Extend post microcode reload to support IBPB feature
    - KVM: SVM: Do not intercept new speculative control MSRs
    - x86/svm: Set IBRS value on VM entry and exit
    - x86/svm: Set IBPB when running a different VCPU
    - KVM: x86: Add speculative control CPUID support for guests
    - SAUCE: turn off IBPB when full retpoline is present

  * Artful 4.13 fixes for tun (LP: #1748846)
    - tun: call dev_get_valid_name() before register_netdevice()
    - tun: allow positive return values on dev_get_valid_name() call
    - tun/tap: sanitize TUNSETSNDBUF input

  * boot failure on AMD Raven + WestonXT (LP: #1742759)
    - SAUCE: drm/amdgpu: add atpx quirk handling (v2)

linux (4.13.0-33.36) artful; urgency=low

  * linux: 4.13.0-33.36 -proposed tracker (LP: #1746903)

  [ Stefan Bader ]
  * starting VMs causing retpoline4 to reboot (LP: #1747507) // CVE-2017-5715
    (Spectre v2 retpoline)
    - x86/retpoline: Fill RSB on context switch for affected CPUs
    - x86/retpoline: Add LFENCE to the retpoline/RSB filling RSB macros
    - x86/retpol...

Changed in linux (Ubuntu Artful):
status: Fix Committed → Fix Released
Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.