upgrading linux-image package to 4.4.0-103.126 breaks Ceph network file system connection

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1737033

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Please note that while the apport report is from a virtualbox VM, this is happening across my entire network. All my workstation are having the same issue, and all are 'fixed' by booting with the previous kernel.

Hello,
for the "previous kernel", do you mean 4.4.0-102 or 4.4.0-101?
Thanks

4.4.0-101 is the one they upgraded from. I'm not seeing 4.4.0.102 anywhere. It's not in the available package list of the machine I'm in front of right now.

Good to know, thanks!
We do have some ceph related change from 4.4.0-101 to 4.4.0-103
https://launchpad.net/ubuntu/+source/linux/4.4.0-103.126

On the running system after you rollback to the previous kernel, can you paste your /etc/fstab for the filesystems that failed to mount and the corresponding lines from mount -v? I assume the issues are with nfshome and officeshare:

cat /etc/fstab | grep -i -e nfshome -e officeshare -e office_share
mount -v | grep -i -e nfshome -e officeshare -e office_share

I built a test kernel with a revert of commit ff467fd. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1737033

Can you test this kernel and see if it resolves this bug?

Hi Joseph,

I'm able to reproduce Benjamin's original issue with kernel 4.4.0-103-generic #126-Ubuntu:

# mount -t ceph <ip>:6789:/ /mnt/mycephfs -o name=admin,secret=<redacted>
mount error 5 = Input/output error

I don't get this problem with 4.4.0-101-generic #124-Ubuntu. I'm working with Jay Vosburgh to test a build of kernel 4.4.0-103.126 without the patches from https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1728739

I forgot to mention that I was still able to reproduce the bug with the kernel you built with a revert of commit ff467fd from http://kernel.ubuntu.com/~jsalisbury/lp1737033

David,

Here is the output you requested:

$ cat /etc/fstab | grep -i -e nfshome -e officeshare -e office_share
Hecate-A:6789,Hecate-B:6789:/nfshome /nfshome ceph name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime,_netdev 0 2
Hecate-A:6789,Hecate-B:6789:/officeshare /mnt/share/Office_Share ceph name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime,_netdev 0 2

$ mount -v | grep -i -e nfshome -e officeshare -e office_share
10.10.2.111:6789,10.10.2.113:6789:/officeshare on /mnt/share/Office_Share type ceph (rw,noatime,name=cephfs,secret=<hidden>,acl)
10.10.2.111:6789,10.10.2.113:6789:/nfshome on /nfshome type ceph (rw,noatime,name=cephfs,secret=<hidden>,acl)

Status changed to 'Confirmed' because the bug affects multiple users.

I tested a kernel without the following commits from https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1728739 and I was able to mount the CephFS filesystem successfully:

http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=f14ca6ac3b198f30ee138f02c3c7d380d165736e

http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=09ade67fc6224440719a94e8c8add556eb036437

http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=c043f509409e380281403f66c54794efbd5d0f02

http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=74771960c7794994fd9067cdd8df0725b6ce75f6

http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=5a03f3043cb617cf2cf5ec9cbd0685d0e86e8b0e

http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=75ee940e71ed7841958cace371a5c79076a0378c

I spoke to the kernel team and their plan is to revert and re-upload (likely next week) to restore functionality.

As soon as the package hits proposed, I'll test it. Will this bug be updated when it's ready?

Hi Benjamin,

Yes we will keep this bug updated, the next SRU cycle is currently underway, the package should be ready in proposed pocket after 17-Dec.

next cycle: 08-Dec through 06-Jan
====================================================================
         08-Dec Last day for kernel commits for this cycle.
11-Dec - 16-Dec Kernel prep week.
17-Dec - 05-Jan Bug verification & Regression testing.
         08-Jan Release to -updates.

I noticed that kernel 4.4.0-104 is in proposed while poking around for something else. I've installed it on my virtualbox test mule and it's working. I can mount Ceph without any issues.

I'm going to test a few things, then push this out to my workstations using my in-house repo. I'll let you know if I have any other issues with it.

After briefly testing to make sure things looked sane, I pushed the update out via our internal repo. No problems so far, but I'll post here if anything comes up.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Please note that we are re-spinning Xenial kernels to release a fix for this issue faster, so the dates from comment #25 don't apply for this bug.

I also confirm the kernel 4.4.0-104.127 in xenial-proposed fixes the issue. I am able to mount my CephFS filesystem normally. Benjamin also confirmed the kernel works for him in comments #26 and #27. I am changing the tag to verification-done-xenial. Thanks!

Thanks @davecore and @benjamin-long for verifying the fix!

This bug was fixed in the package linux - 4.4.0-104.127

---------------
linux (4.4.0-104.127) xenial; urgency=low

  * linux: 4.4.0-104.127 -proposed tracker (LP: #1737511)

  * upgrading linux-image package to 4.4.0-103.126 breaks Ceph network file
    system connection (LP: #1737033)
    - Revert "libceph: MOSDOpReply v7 encoding"
    - Revert "libceph: advertise support for TUNABLES5"
    - Revert "crush: decode and initialize chooseleaf_stable"
    - Revert "crush: add chooseleaf_stable tunable"
    - Revert "crush: ensure take bucket value is valid"
    - Revert "crush: ensure bucket id is valid before indexing buckets array"

 -- Kleber Sacilotto de Souza <email address hidden> Mon, 11 Dec 2017 12:20:36 +0100