bionic, xenial/hwe: misses "fuse: fix initial parallel dirops" patch

Bug #1823972 reported by Kirill Smelkov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
Undecided
Andrea Righi
Bionic
Fix Released
Undecided
Unassigned

Bug Description

SRU Justification:

[Impact]

 * Enabling parallel dirops in fuse (FUSE_PARALLEL_DIROPS) may cause a race condition that leave fuse inode's mutex held, triggering a deadlock

 * The problem is that the lock and unlock paths are relying on get_fuse_conn(inode)->parallel_dirops to decide if the mutex needs to be acquired/released, but its value might be set in the lock path and unset in the unlock path (leaving the mutex held)

[Test Case]

 * A test case that triggers the bug almost immediately can be found here https://github.com/hanwen/go-fuse/pull/288

[Fix]

 * Instead of relying on get_fuse_conn(inode)->parallel_dirops both in fuse_lock_inode() and fuse_unlock_inode(), only check this flag in the locking path and pass a variable to fuse_unlock_inode() to determine if the mutex was acquired or not

[Regression Potential]

 * Fix has been tested on the affected platform. It is an upstream fix that seems to affect only 4.7+ kernels, more exactly in our case only Bionic kernels (and derived) are affected. Cosmic and above already include this fix. So regression potential is minimal.

[Original bug report]

Hello up there,

We were reported about a deadlock in the kernel while using a FUSE-based filesystem on Ubuntu.
The kernel in question is Ubuntu-hwe-4.15.0-47.50~16.04.1 from Xenial/HWE. We tracked this bug to the fact that 4.15.x kernel in Ubuntu does not include the following patch, in despite the patch being marked as needed for v4.7+ stable kernels:

https://git.kernel.org/linus/63576c13bd

Please see the following go-fuse issue for full details:

https://github.com/hanwen/go-fuse/issues/281#issuecomment-480008562

The bug is potentially applicable to libfuse users too since libfuse by default enables parallel dirops whenever kernel claims support for it, which libfuse maintained confirmed:

https://github.com/hanwen/go-fuse/issues/281#issuecomment-480013202
https://github.com/hanwen/go-fuse/issues/281#issuecomment-480510381

We tested that cherry-picking 63576c13bd into 4.15.x series makes the problem go away:

https://github.com/hanwen/go-fuse/issues/281#issuecomment-480499969

So please include https://git.kernel.org/linus/63576c13bd into Ubuntu 4.15.x kernel series which are bionic/master and xenial/hwe, and which currently don't have this patch.

--------

Here is a full list of FUSE patches marked to be needed in stable kernels starting from v4.15:

kirr@deco:~/src/linux/linux$ git log --oneline v4.15..v5.1-rc3 --grep="stable@" -- fs/fuse/
a2ebba824106 fuse: decrement NR_WRITEBACK_TEMP on the right page
9509941e9c53 fuse: call pipe_buf_release() under pipe lock
8a3177db59cd cuse: fix ioctl
97e1532ef81a fuse: handle zero sized retrieve correctly
2e64ff154ce6 fuse: continue to send FUSE_RELEASEDIR when FUSE_OPEN returns ENOSYS
ebacb8127359 fuse: fix use-after-free in fuse_direct_IO()
2d84a2d19b61 fuse: fix possibly missed wake-up after abort
7fabaf303458 fuse: fix leaked notify reply
908a572b80f6 fuse: fix blocked_waitq wakeup
4c316f2f3ff3 fuse: set FR_SENT while locked
d2d2d4fb1f54 fuse: Fix use-after-free in fuse_dev_do_write()
bc78abbd55dd fuse: Fix use-after-free in fuse_dev_do_read()
a2477b0e67c5 fuse: Don't access pipe->buffers without pipe_lock()
63576c13bd17 fuse: fix initial parallel dirops
e8f3bd773d22 fuse: Fix oops at process_init_reply()
b8f95e5d13f5 fuse: umount should wait for all requests
45ff350bbd9d fuse: fix unlocked access to processing queue
87114373ea50 fuse: fix double request_end()
543b8f8662fe (tag: fuse-update-4.18) fuse: don't keep dead fuse_conn at fuse_fill_super().
6becdb601bae fuse: fix control dir setup and teardown
8a301eb16d99 fuse: fix congested state leak on aborted connections
df0e91d48827 fuse: atomic_o_trunc should truncate pagecache

Among those only 8a3177db59cd and 2d84a2d19b61 should not be applied to 4.15.x becuase they cure a problem introduced in a later kernel (please see got log without --oneline for stable@ details)

However both bionic and xenial/hwe has much less fuse patches applied:

kirr@deco:~/src/linux/linux$ git log --oneline v4.15..bionic/master -- fs/fuse/
e992e3521885 fuse: fix control dir setup and teardown
f3a3e0537dcd fuse: don't keep dead fuse_conn at fuse_fill_super().
840c77082f93 fuse: atomic_o_trunc should truncate pagecache
c0e31b214498 fuse: fix congested state leak on aborted connections
45f23c59120f UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts
1223588451c6 UBUNTU: SAUCE: (namespace) fuse: Restrict allow_other to the superblock's namespace or a descendant
b4d1889491a0 UBUNTU: SAUCE: (namespace) fuse: Support fuse filesystems outside of init_user_ns

kirr@deco:~/src/linux/linux$ git log --oneline v4.15..xenial/hwe -- fs/fuse/
e992e3521885 fuse: fix control dir setup and teardown
f3a3e0537dcd fuse: don't keep dead fuse_conn at fuse_fill_super().
840c77082f93 fuse: atomic_o_trunc should truncate pagecache
c0e31b214498 fuse: fix congested state leak on aborted connections
45f23c59120f UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts
1223588451c6 UBUNTU: SAUCE: (namespace) fuse: Restrict allow_other to the superblock's namespace or a descendant
b4d1889491a0 UBUNTU: SAUCE: (namespace) fuse: Support fuse filesystems outside of init_user_ns

which suggests that other FUSE fixes should be cherry-picked too.

Please consider cherry-picking those additional patches too. They are all in upstream kernel
stable series, e.g. stable/linux-4.14.y has them:

kirr@deco:~/src/linux/linux$ git log --oneline v4.14..stable/linux-4.14.y -- fs/fuse/
266a69895b89 fuse: handle zero sized retrieve correctly
b928e93d864c fuse: decrement NR_WRITEBACK_TEMP on the right page
65f222bb370e fuse: call pipe_buf_release() under pipe lock
c1149b873482 fuse: continue to send FUSE_RELEASEDIR when FUSE_OPEN returns ENOSYS
6ceec07cc84a fuse: fix leaked notify reply
a42d933dc281 fuse: fix use-after-free in fuse_direct_IO()
78da72ee42d8 fuse: set FR_SENT while locked
f6f21a2b70c6 fuse: fix blocked_waitq wakeup
ab962e91008a fuse: Fix use-after-free in fuse_dev_do_write()
d94b3a2375cb fuse: Fix use-after-free in fuse_dev_do_read()
e8a3f3a03655 fuse: Add missed unlock_page() to fuse_readpages_fill()
ff4a71855d0a fuse: Fix oops at process_init_reply()
973206923812 fuse: umount should wait for all requests
fc17d7519e8e fuse: fix unlocked access to processing queue
cfb6eca6e4bb fuse: fix double request_end()
7d392674443c fuse: fix initial parallel dirops
eaebcf902ae0 fuse: Don't access pipe->buffers without pipe_lock()
69829f749a43 fuse: fix control dir setup and teardown
3a37d85a90da fuse: don't keep dead fuse_conn at fuse_fill_super().
2f7bf369b5f8 fuse: atomic_o_trunc should truncate pagecache
02832578eb9d fuse: fix congested state leak on aborted connections

and it is just that stable/linux-4.15.y stopped being maintained by Greg KH.

Thanks beforehand,
Kirill
---
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDesktop: Unity
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=UUID=dec25862-42c7-4494-8f75-e6cc76aa65ea
InstallationDate: Installed on 2019-02-28 (39 days ago)
InstallationMedia: Ubuntu 16.04.6 LTS "Xenial Xerus" - Release amd64 (20190227)
IwConfig:
 lo no wireless extensions.

 ens3 no wireless extensions.
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
ProcFB: 0 virtiodrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-45-generic root=UUID=fe1f50d9-7142-4ee3-8904-bae7d80bc6b1 ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 4.15.0-45.48~16.04.1-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-45-generic N/A
 linux-backports-modules-4.15.0-45-generic N/A
 linux-firmware 1.157.21
RfKill:

Tags: xenial
Uname: Linux 4.15.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.12.0-1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-3.1
dmi.modalias: dmi:bvnSeaBIOS:bvr1.12.0-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-3.1:cvnQEMU:ct1:cvrpc-i440fx-3.1:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-3.1
dmi.sys.vendor: QEMU

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1823972

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: xenial
Revision history for this message
Kirill Smelkov (navytux) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected
description: updated
Revision history for this message
Kirill Smelkov (navytux) wrote : CRDA.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : Lspci.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : ProcEnviron.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : ProcModules.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : PulseList.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : UdevDb.txt

apport information

Revision history for this message
Kirill Smelkov (navytux) wrote : WifiSyslog.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Kirill Smelkov (navytux)
tags: added: bionic
Andrea Righi (arighi)
description: updated
Changed in linux (Ubuntu):
assignee: nobody → Andrea Righi (arighi)
description: updated
Revision history for this message
Andrea Righi (arighi) wrote :

Sent an SRU fix to include 63576c13bd in Bionic.

Revision history for this message
Kirill Smelkov (navytux) wrote :

Andrea, thanks.

What about other fs/fuse/ patches marked as @stable? We have more problems reported on 4.15.x Ubuntu kernel (see e.g. https://github.com/hanwen/go-fuse/issues/287) and that is unfortunate when the fix/patch is known, marked as needed to be applied to stable kernel, but Ubuntu is not picking those patches up, just because upstream 4.15 kernel is not LTS release and stopped being updated long ago, unlike e.g. upstream 4.4 and 4.14 stable releases which are updated to this day and have all those stable patches I'm talking about.

Revision history for this message
Andrea Righi (arighi) wrote :

I'll take a look at the other patches, I think they can be easily applied to 4.15 if they don't require too much backporting work.

Revision history for this message
Kirill Smelkov (navytux) wrote :

Thanks. If I understood correctly, the patches should be more or less directly applicable to 4.15.y

Revision history for this message
Andrea Righi (arighi) wrote :

@navytux yes, a few of them require minimal changes, but most of them can be applied cleanly to 4.15.

Revision history for this message
Andrea Righi (arighi) wrote :

All fixes applied here:

 git://git.launchpad.net/~arighi/+git/bionic-linux fuse-fixes

And also sent an SRU pull request. Let's see if they get merged.

Revision history for this message
Kirill Smelkov (navytux) wrote :

@arighi, thanks. Let me know if you need more testing and/or help. Good luck with the review.

Changed in linux (Ubuntu Bionic):
status: New → Fix Committed
Revision history for this message
Kirill Smelkov (navytux) wrote :

@arighi, @kmously, thanks for pulling all fuse patches to bionic/master-next. Am I right in my understanding that the patches will also propagate to xenial/hwe eventually?

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
Jakob Unterwurzacher (jakobunt) wrote :

$ apt-cache policy linux-generic-hwe-16.04
linux-generic-hwe-16.04:
  Installed: 4.15.0.49.70
  Candidate: 4.15.0.49.70
  Version table:
 *** 4.15.0.49.70 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     4.15.0.48.69 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages

$ uname -a
Linux osboxes 4.15.0-49-generic #52~16.04.1-Ubuntu SMP Thu Apr 25 18:54:26 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Issue is fixed.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (12.6 KiB)

This bug was fixed in the package linux - 4.15.0-50.54

---------------
linux (4.15.0-50.54) bionic; urgency=medium

  * CVE-2018-12126 // CVE-2018-12127 // CVE-2018-12130
    - Documentation/l1tf: Fix small spelling typo
    - x86/cpu: Sanitize FAM6_ATOM naming
    - kvm: x86: Report STIBP on GET_SUPPORTED_CPUID
    - locking/atomics, asm-generic: Move some macros from <linux/bitops.h> to a
      new <linux/bits.h> file
    - tools include: Adopt linux/bits.h
    - x86/msr-index: Cleanup bit defines
    - x86/speculation: Consolidate CPU whitelists
    - x86/speculation/mds: Add basic bug infrastructure for MDS
    - x86/speculation/mds: Add BUG_MSBDS_ONLY
    - x86/kvm: Expose X86_FEATURE_MD_CLEAR to guests
    - x86/speculation/mds: Add mds_clear_cpu_buffers()
    - x86/speculation/mds: Clear CPU buffers on exit to user
    - x86/kvm/vmx: Add MDS protection when L1D Flush is not active
    - x86/speculation/mds: Conditionally clear CPU buffers on idle entry
    - x86/speculation/mds: Add mitigation control for MDS
    - x86/speculation/mds: Add sysfs reporting for MDS
    - x86/speculation/mds: Add mitigation mode VMWERV
    - Documentation: Move L1TF to separate directory
    - Documentation: Add MDS vulnerability documentation
    - x86/speculation/mds: Add mds=full,nosmt cmdline option
    - x86/speculation: Move arch_smt_update() call to after mitigation decisions
    - x86/speculation/mds: Add SMT warning message
    - x86/speculation/mds: Fix comment
    - x86/speculation/mds: Print SMT vulnerable on MSBDS with mitigations off
    - x86/speculation/mds: Add 'mitigations=' support for MDS

  * CVE-2017-5715 // CVE-2017-5753
    - s390/speculation: Support 'mitigations=' cmdline option

  * CVE-2017-5715 // CVE-2017-5753 // CVE-2017-5754 // CVE-2018-3639
    - powerpc/speculation: Support 'mitigations=' cmdline option

  * CVE-2017-5715 // CVE-2017-5754 // CVE-2018-3620 // CVE-2018-3639 //
    CVE-2018-3646
    - cpu/speculation: Add 'mitigations=' cmdline option
    - x86/speculation: Support 'mitigations=' cmdline option

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

linux (4.15.0-49.53) bionic; urgency=medium

  * linux: 4.15.0-49.53 -proposed tracker (LP: #1826358)

  * Backport support for software count cache flush Spectre v2 mitigation. (CVE)
    (required for POWER9 DD2.3) (LP: #1822870)
    - powerpc/64s: Add support for ori barrier_nospec patching
    - powerpc/64s: Patch barrier_nospec in modules
    - powerpc/64s: Enable barrier_nospec based on firmware settings
    - powerpc: Use barrier_nospec in copy_from_user()
    - powerpc/64: Use barrier_nospec in syscall entry
    - powerpc/64s: Enhance the information in cpu_show_spectre_v1()
    - powerpc/64: Disable the speculation barrier from the command line
    - powerpc/64: Make stf barrier PPC_BOOK3S_64 specific.
    - powerpc/64: Add CONFIG_PPC_BARRIER_NOSPEC
    - powerpc/64: Call setup_barrier_nospec() from setup_arch()
    - powerpc/64: Make meltdown reporting Book3S 64 specific
    - powerpc/lib/code-patching: refactor patch_instruction()
    - powerpc/lib/feature-fixups: use raw_patch_instruction()
    - powerpc/asm: Add a patch_site mac...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of the Stable Release Update for linux-aws has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Kirill Smelkov (navytux) wrote :

Thanks for fixing this issue. We confirm that the problem is gone with updated kernel.

Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu):
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.