Invalid btree pointer causes the kernel NULL pointer dereference

Bug #1729256 reported by Gavin Guo
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Gavin Guo
Xenial
Fix Released
Medium
Unassigned
Zesty
Fix Released
Medium
Unassigned
Artful
Won't Fix
Medium
Unassigned

Bug Description

[Impact]
The frequent kernel errors can be seen inside the XFS based OSD
processes and it causes to crash and restart.

BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0
IP: [<ffffffffc03705a0>] xfs_da3_node_read+0x30/0xb0 [xfs]
CPU: 8 PID: 2855031 Comm: tp_fstore_op Not tainted 4.4.0-78-generic #99-Ubuntu
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 09/13/2016
task: ffff880620f38e00 ti: ffff880678228000 task.ti: ffff880678228000
RIP: 0010:[<ffffffffc03705a0>] [<ffffffffc03705a0>]
xfs_da3_node_read+0x30/0xb0 [xfs]
RSP: 0018:ffff88067822bd00 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffff88078abff3f0 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88067822bcb0
RBP: ffff88067822bd20 R08: 0000000000000001 R09: fffffffffffffffe
R10: ffffea001e2aff80 R11: 0000000000000001 R12: ffff88067822bd50
R13: ffff8805a37f0000 R14: 0000000000000001 R15: 000000008d180e3e
FS: 00007f7573c01700(0000) GS:ffff88103fa00000(0000)knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000a0 CR3: 0000001686f3d000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 ffffffffc03cfb50 ffffffffc03b0ecc ffff88067822bde0 0000000000000001
 ffff88067822bd98 ffffffffc038c8b3 0000000200000008 ffff88081cb06040
 00000002d8723bf8 ffff880fc142ae80 0000000000000000 0000000000000000
Call Trace:
 [<ffffffffc03b0ecc>] ? xfs_trans_roll+0x2c/0x50 [xfs]
 [<ffffffffc038c8b3>] xfs_attr3_node_inactive+0x183/0x220 [xfs]
 [<ffffffffc038c9fc>] xfs_attr3_root_inactive+0xac/0x100 [xfs]
 [<ffffffffc038cb9c>] xfs_attr_inactive+0x14c/0x1a0 [xfs]
 [<ffffffffc03a6da5>] xfs_inactive+0x85/0x120 [xfs]
 [<ffffffffc03ac2f5>] xfs_fs_evict_inode+0xa5/0x100 [xfs]
 [<ffffffff8122aaee>] evict+0xbe/0x190
 [<ffffffff8122add1>] iput+0x1c1/0x240
 [<ffffffff8121f859>] do_unlinkat+0x199/0x2d0
 [<ffffffff812203f6>] SyS_unlink+0x16/0x20
 [<ffffffff81840a32>] entry_SYSCALL_64_fastpath+0x16/0x71

[Fix]
commit e678a63e6c95f140befe6fcd81b49075ecb3c701
Author: Brian Foster <email address hidden>
Date: Mon Oct 9 11:38:56 2017 -0700

xfs: reinit btree pointer on attr tree inactivation walk

xfs_attr3_root_inactive() walks the attr fork tree to invalidate the
associated blocks. xfs_attr3_node_inactive() recursively descends from
internal blocks to leaf blocks, caching block address values along the
way to revisit parent blocks, locate the next entry and descend down
that branch of the tree.

The code that attempts to reread the parent block is unsafe because it
assumes that the local xfs_da_node_entry pointer remains valid after
an xfs_trans_brelse() and re-read of the parent buffer. Under heavy
memory pressure, it is possible that the buffer has been reclaimed and
reallocated by the time the parent block is reread. This means that
'btree' can point to an invalid memory address, lead to a
random/garbage value for child_fsb and cause the subsequent read of
the attr fork to go off the rails and return a NULL buffer for an attr
fork offset that is most likely not allocated.

Note that this problem can be manufactured by setting
XFS_ATTR_BTREE_REF to 0 to prevent LRU caching of attr buffers,
creating a file with a multi-level attr fork and removing it to
trigger inactivation.

To address this problem, reinit the node/btree pointers to the parent
buffer after it has been re-read. This ensures btree points to a valid
record and allows the walk to proceed.

[Test]
The patch has been tested on the production system.

[Regression Risk]
Clean cherry-pick queued for stable, so we'll be picking it up anyway.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1729256

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: xenial
Gavin Guo (mimi0213kimo)
description: updated
description: updated
description: updated
Revision history for this message
Stefan Bader (smb) wrote :

In Artful this fix was commited as part of the 4.13.10 upstream stable set.

Changed in linux (Ubuntu Xenial):
importance: Undecided → Medium
status: New → Fix Committed
Changed in linux (Ubuntu Zesty):
importance: Undecided → Medium
status: New → Fix Committed
Changed in linux (Ubuntu Artful):
importance: Undecided → Medium
status: New → Fix Committed
Revision history for this message
Khaled El Mously (kmously) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
tags: added: verification-needed-zesty
Revision history for this message
Khaled El Mously (kmously) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-zesty' to 'verification-done-zesty'. If the problem still exists, change the tag 'verification-needed-zesty' to 'verification-failed-zesty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Gavin Guo (mimi0213kimo)
tags: added: verification-done-xenial verification-done-zesty
removed: verification-needed-xenial verification-needed-zesty
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 4.10.0-42.46

---------------
linux (4.10.0-42.46) zesty; urgency=low

  * linux: 4.10.0-42.46 -proposed tracker (LP: #1736152)

  * CVE-2017-1000405
    - mm, thp: Do not make page table dirty unconditionally in touch_p[mu]d()

  * CVE-2017-16939
    - ipsec: Fix aborted xfrm policy dump crash

linux (4.10.0-41.45) zesty; urgency=low

  * linux: 4.10.0-41.45 -proposed tracker (LP: #1733524)

  * tar -x sometimes fails on overlayfs (LP: #1728489)
    - ovl: check if all layers are on the same fs
    - ovl: persistent inode number for directories

  * CVE-2017-12146
    - driver core: platform: fix race condition with driver_override

  * NVMe timeout is too short (LP: #1729119)
    - nvme: update timeout module parameter type

  * Set PANIC_TIMEOUT=10 on Power Systems (LP: #1730660)
    - [Config]: Set PANIC_TIMEOUT=10 on ppc64el

  * Cannot pair BLE remote devices when using combo BT SoC (LP: #1731467)
    - Bluetooth: increase timeout for le auto connections

  * Plantronics P610 does not support sample rate reading (LP: #1719853)
    - ALSA: usb-audio: Add sample rate quirk for Plantronics P610

  * Invalid btree pointer causes the kernel NULL pointer dereference
    (LP: #1729256)
    - xfs: reinit btree pointer on attr tree inactivation walk

  * Samba mount/umount in docker container triggers kernel Oops (LP: #1729637)
    - ipv6: only call ip6_route_dev_notify() once for NETDEV_UNREGISTER
    - ipv6: fix NULL dereference in ip6_route_dev_notify()

  * Device hotplugging with MPT SAS cannot work for VMWare ESXi (LP: #1730852)
    - scsi: mptsas: Fixup device hotplug for VMWare ESXi

  * Boot/Installation crash of Ubuntu-16.04.3 HWE kernel on R940 (LP: #1719697)
    - Revert "x86/acpi: Set persistent cpuid <-> nodeid mapping when booting"

 -- Stefan Bader <email address hidden> Mon, 04 Dec 2017 15:04:01 +0100

Changed in linux (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (9.5 KiB)

This bug was fixed in the package linux - 4.4.0-103.126

---------------
linux (4.4.0-103.126) xenial; urgency=low

  * linux: 4.4.0-103.126 -proposed tracker (LP: #1736181)

  * CVE-2017-1000405
    - mm, thp: Do not make page table dirty unconditionally in touch_p[mu]d()

  * CVE-2017-16939
    - netlink: add a start callback for starting a netlink dump
    - ipsec: Fix aborted xfrm policy dump crash

linux (4.4.0-102.125) xenial; urgency=low

  * linux: 4.4.0-102.125 -proposed tracker (LP: #1733541)

  * tar -x sometimes fails on overlayfs (LP: #1728489)
    - ovl: check if all layers are on the same fs
    - ovl: persistent inode number for directories

  * NVMe timeout is too short (LP: #1729119)
    - nvme: update timeout module parameter type

  * Set PANIC_TIMEOUT=10 on Power Systems (LP: #1730660)
    - [Config]: Set PANIC_TIMEOUT=10 on ppc64el

  * Cannot pair BLE remote devices when using combo BT SoC (LP: #1731467)
    - Bluetooth: increase timeout for le auto connections

  * CIFS errors on 4.4.0-98, but not on 4.4.0-97 with same config (LP: #1729337)
    - SMB3: Validate negotiate request must always be signed

  * Plantronics P610 does not support sample rate reading (LP: #1719853)
    - ALSA: usb-audio: Add sample rate quirk for Plantronics P610

  * Invalid btree pointer causes the kernel NULL pointer dereference
    (LP: #1729256)
    - xfs: reinit btree pointer on attr tree inactivation walk

  * Samba mount/umount in docker container triggers kernel Oops (LP: #1729637)
    - ipv6: only call ip6_route_dev_notify() once for NETDEV_UNREGISTER
    - ipv6: fix NULL dereference in ip6_route_dev_notify()

  * [kernel] tty/hvc: Use opal irqchip interface if available (LP: #1728098)
    - tty/hvc: Use opal irqchip interface if available

  * Device hotplugging with MPT SAS cannot work for VMWare ESXi (LP: #1730852)
    - scsi: mptsas: Fixup device hotplug for VMWare ESXi

  * NMI watchdog: BUG: soft lockup on Guest upon boot (KVM) (LP: #1727331)
    - KVM: PPC: Book3S: Treat VTB as a per-subcore register, not per-thread

  * Attempt to map rbd image from ceph jewel/luminous hangs (LP: #1728739)
    - crush: ensure bucket id is valid before indexing buckets array
    - crush: ensure take bucket value is valid
    - crush: add chooseleaf_stable tunable
    - crush: decode and initialize chooseleaf_stable
    - libceph: advertise support for TUNABLES5
    - libceph: MOSDOpReply v7 encoding

  * Xenial update to 4.4.98 stable release (LP: #1732698)
    - adv7604: Initialize drive strength to default when using DT
    - video: fbdev: pmag-ba-fb: Remove bad `__init' annotation
    - PCI: mvebu: Handle changes to the bridge windows while enabled
    - xen/netback: set default upper limit of tx/rx queues to 8
    - drm: drm_minor_register(): Clean up debugfs on failure
    - KVM: PPC: Book 3S: XICS: correct the real mode ICP rejecting counter
    - iommu/arm-smmu-v3: Clear prior settings when updating STEs
    - powerpc/corenet: explicitly disable the SDHC controller on kmcoge4
    - ARM: omap2plus_defconfig: Fix probe errors on UARTs 5 and 6
    - crypto: vmx - disable preemption to enable vsx in aes_ctr.c
    - iio: trigger: free trigger...

Read more...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Andy Whitcroft (apw) wrote : Closing unsupported series nomination.

This bug was nominated against a series that is no longer supported, ie artful. The bug task representing the artful nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu Artful):
status: Fix Committed → Won't Fix
Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu):
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.