btrfs and tar sparse truncate archives

This change was made by a bot.

I'd like to perform a "Reverse" bisect to figure out what commit fixes this bug. We need to identify the last kernel version that had the bug, and the first kernel version that fixed the bug.

Can you test the following kernels and report back:

v4.14-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc1/
v4.14 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14
v4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/

You don't have to test every kernel, just up until the first kernel that does not have the bug.

Thanks in advance!

v4.14-rc1: BAD

ubuntu@ubuntu:~$ uname -r
4.14.0-041400rc1-generic
ubuntu@ubuntu:~$ sudo bash
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
0ce8c4139740198926273853defcb12a -

v4.14 Final: BAD

root@ubuntu:~# uname -r
4.14.0-041400-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
0ce8c4139740198926273853defcb12a -

v4.15-rc1: BAD

root@ubuntu:~# uname -r
4.15.0-041500rc1-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
0ce8c4139740198926273853defcb12a -

Bionic v4.15.0-12-generic: GOOD
root@ubuntu:~# uname -r
4.15.0-12-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

The Bionic 4.15.0-12 kernel is based off of upstream 4.15.7. Can you test that kernel as well? That will tell us if the fix in Bionic is an Ubuntu SAUCE patch or a 4.15 stable update.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15.7/

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=bd40234f

v4.15.7: GOOD

root@ubuntu:~# uname -r
4.15.7-041507-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

@Scott, will that upstream commit make it so the kernel fix we are looking for is not required?

On Thu, Mar 22, 2018 at 2:56 PM, Joseph Salisbury
<email address hidden> wrote:
> @Scott, will that upstream commit make it so the kernel fix we are
> looking for is not required?

This is a curtin workaround; we're not using the sparse flag when invoking tar;
However, the bug/issue remains between tar and btrfs.

Outside of curtin's use of it it, any users of btrfs with any "sparse"
file detection are
vulnerable.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1757565
>
> Title:
> btrfs and tar sparse truncate archives
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757565/+subscriptions

Thanks for the info, Ryan.

We know that 4.15-rc1 is BAD and 4.15.7 is GOOD, so the fix came in some where in between.

Can you next test v4.15 Final:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15/

v4.15 Final: GOOD

root@ubuntu:~# uname -r
4.15.0-041500-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

That would indicate the fix is in from v4.15-rc2 -> 4.15 Final. I'll review the commits in that range, but there are probably thousands of them. It will help to narrow it down further. Could you test the 4.15 release candidates on the following page to find the last good kernel and first bad kernel:

http://kernel.ubuntu.com/~kernel-ppa/mainline/

v4.15-rc6: GOOD

root@ubuntu:~# uname -r
4.15.0-041500rc6-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

v4.15-rc4: GOOD

root@ubuntu:~# uname -r
4.15.0-041500rc4-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

v4.15-rc3: GOOD

root@ubuntu:~# uname -r
4.15.0-041500rc3-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

v4.14-rc2: GOOD

root@ubuntu:~# uname -r
4.15.0-041500rc2-generic
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

So, somewhere between rc1 and rc2.

This looks promising and landed in -rc2:

Btrfs: fix reported number of inode blocks after buffered append writes
https://www.spinics.net/lists/linux-btrfs/msg71274.html

Thanks for doing all the leg work. I'll build a test kernel with that commit and post it shortly.

It looks like that commit landed in -rc3:

git describe --contains e3b8a48
v4.15-rc3~3^2~11

However, -rc2 is good, correct? I'll have a look at the other btrfs commits in -rc2 and see if any stick out.

Hmm, even though git describe returns that tag, it seem s-rc2 has that commit.

I'll still build an Artful test kernel with that commit to see if it fixes the bug and figure out the tagging in the background.

Yeah, rc2 was fine.

I built an Artful test kernel with commit e3b8a48. It also needed commit f48bf66b6 as a prereq.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1757565

Can you test this kernel and see if it resolves this bug?

Note, to test this kernel, you need to install both the linux-image and linux-image-extra .deb packages.

Thanks in advance!

Test kernel: GOOD

root@ubuntu:~# cat /proc/version_signature
Ubuntu 4.13.0-37.42~lp1757565-generic 4.13.13
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
e4121d2f3126b8c364bfa1aaa82371a3 -

That's great news. I'll SRU those two commits to the Artful kernel.

SRU request submitted:
https://lists.ubuntu.com/archives/kernel-team/2018-March/090976.html

Joseph,
Thanks for providing the link to the mailing list post. That is very
helpful.

On Fri, Mar 23, 2018 at 4:15 PM, Joseph Salisbury <
<email address hidden>> wrote:

> SRU request submitted:
> https://lists.ubuntu.com/archives/kernel-team/2018-March/090976.html
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1757565
>
> Title:
> btrfs and tar sparse truncate archives
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1757565/+subscriptions
>

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Verified artful-proposed.

root@ubuntu:~# cat /etc/cloud/build.info
build_name: server
serial: 20180404
root@ubuntu:~# uname -a
Linux ubuntu 4.13.0-38-generic #43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:~# mount /dev/sda /mnt
root@ubuntu:~# grep sda /proc/mounts
/dev/sda /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
root@ubuntu:~# SPARSE="-S"; rm -rf /mnt/tmp; md5sum /usr/bin/python3.6; mkdir -p /mnt/tmp; cp -a /usr/bin/python3.6 /mnt/tmp; tar -C /mnt/tmp $SPARSE -czf /mnt/test.tgz .; tar $SPARSE -xzf /mnt/test.tgz -O | md5sum
e4121d2f3126b8c364bfa1aaa82371a3 /usr/bin/python3.6
0ce8c4139740198926273853defcb12a -
root@ubuntu:~# dpkg --list | grep virtual
ii linux-headers-virtual 4.13.0.38.41 amd64 Virtual Linux kernel headers
ii linux-image-virtual 4.13.0.38.41 amd64 Virtual Linux kernel image
ii linux-virtual 4.13.0.38.41 amd64 Minimal Generic Linux kernel and headers
ii open-vm-tools 2:10.1.10-3ubuntu0.1 amd64 Open VMware Tools for virtual machines hosted on VMware (CLI)
root@ubuntu:~# apt update && apt install linux-image-virtual
Hit:1 http://archive.ubuntu.com/ubuntu artful InRelease
Hit:2 http://security.ubuntu.com/ubuntu artful-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu artful-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu artful-backports InRelease
Get:5 http://archive.ubuntu.com/ubuntu artful-proposed InRelease [235 kB]
Get:6 http://archive.ubuntu.com/ubuntu artful-proposed/main amd64 Packages [62.3 kB]
Get:7 http://archive.ubuntu.com/ubuntu artful-proposed/main Translation-en [26.1 kB]
Fetched 323 kB in 1s (206 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
29 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  linux-headers-4.13.0-39 linux-headers-4.13.0-39-generic linux-headers-generic linux-headers-virtual linux-image-4.13.0-39-generic
  linux-virtual
Suggested packages:
  fdutils linux-doc-4.13.0 | linux-source-4.13.0 linux-tools
The following NEW packages will be installed:
  linux-headers-4.13.0-39 linux-headers-4.13.0-39-generic linux-image-4.13.0-39-generic
The following packages will be upgraded:
  linux-headers-generic linux-headers-virtual linux-image-virtual linux-virtual
4 upgraded, 3 newly installed, 0 to remove and 25 not upgraded.
Need to get 32.5 MB of archives.
After this operation, 156 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu artful-proposed/main amd64 linux-headers-4.13.0-39 all 4.13.0-39.44 [10.9 MB]
Get:2 http://archive.ubuntu.com/ubuntu artful-proposed/ma...

This bug was fixed in the package linux - 4.13.0-39.44

---------------
linux (4.13.0-39.44) artful; urgency=medium

  * linux: 4.13.0-39.44 -proposed tracker (LP: #1761456)

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2
    Intel) // CVE-2017-5754
    - x86/mm: Reinitialize TLB state on hotplug and resume

  * intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-
    image-4.13.0-37-generic) (LP: #1759920) // CVE-2017-5715 (Spectre v2 Intel)
    - Revert "x86/mm: Only set IBPB when the new thread cannot ptrace current
      thread"
    - x86/speculation: Use Indirect Branch Prediction Barrier in context switch

  * DKMS driver builds fail with: Cannot use CONFIG_STACK_VALIDATION=y, please
    install libelf-dev, libelf-devel or elfutils-libelf-devel (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

  * retpoline hints: primary infrastructure and initial hints (LP: #1758856)
    - [Packaging] retpoline-extract: flag *0xNNN(%reg) branches
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool
    - x86/speculation, objtool: Annotate indirect calls/jumps for objtool on 32bit
    - x86/paravirt, objtool: Annotate indirect calls
    - [Packaging] retpoline -- add safe usage hint support
    - [Packaging] retpoline-check -- only report additions
    - [Packaging] retpoline -- widen indirect call/jmp detection
    - [Packaging] retpoline -- elide %rip relative indirections
    - [Packaging] retpoline -- clear hint information from packages
    - KVM: x86: Make indirect calls in emulator speculation safe
    - KVM: VMX: Make indirect call speculation safe
    - x86/boot, objtool: Annotate indirect jump in secondary_startup_64()
    - SAUCE: early/late -- annotate indirect calls in early/late initialisation
      code
    - SAUCE: vga_set_mode -- avoid jump tables
    - [Config] retpoline -- switch to new format
    - [Packaging] retpoline hints -- handle missing files when RETPOLINE not
      enabled
    - [Packaging] final-checks -- remove check for empty retpoline files

  * retpoline: ignore %cs:0xNNN constant indirections (LP: #1752655)
    - [Packaging] retpoline -- elide %cs:0xNNNN constants on i386

  * zfs system process hung on container stop/delete (LP: #1754584)
    - SAUCE: Fix non-prefaulted page deadlock (LP: #1754584)

  * zfs-linux 0.6.5.11-1ubuntu5 ADT test failure with linux 4.15.0-1.2
    (LP: #1737761)
    - SAUCE: (noup) Update zfs to 0.6.5.11-1ubuntu3.2

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * btrfs and tar sparse truncate archives (LP: #1757565)
    - Btrfs: move definition of the function btrfs_find_new_delalloc_bytes
    - Btrfs: fix reported number of inode blocks after buffered append writes

  * efifb broken on ThunderX-based Gigabyte nodes (LP: #1758375)
    - drivers/fbdev/efifb: Allow BAR to be moved instead of claiming it

  * Intel i40e PF reset due to incorrect MDD detection (continues...)
    (LP: #1723127)
    - i40e/i40ev...