[quincy] invalid osd_class_dir blocks rados client connections

Status changed to 'Confirmed' because the bug affects multiple users.

While attempting to recreate this issue, we noticed a new Jammy/Quincy installation will load the correct library files!

The freshly deployed Quincy reports the incorrect `osd_class_dir` (lib/x86_64-linux-gnu/rados-classes), but lsof shows the ceph-osd process is loading rados-classes libraries from /usr/lib:
ceph-osd 735836 64045 mem REG 8,2 530688 35521076 /usr/lib/x86_64-linux-gnu/rados-classes/libcls_rbd.so.1.0.0

How is this happening?

It turns out that the correct library file is loaded by accident. New Jammy installations have /lib symlinked to /usr/lib. The ceph-osd processes have a working directory of "/" so the relative "lib/" path follows the symlink to the correct "/usr/lib" location.

Starting with 19.04, merged-usr is now the default for new installations [0]. This merged-usr feature [1] is what creates the symlink from /lib -> /usr/lib.

The key thing to note here is that these symlinks do not get created on hosts that have been upgraded:
"Existing systems, upon upgrade, will not be reconfigured for merged-usr."

Hosts without the /lib -> /usr/lib symlink are exposed to the issue described in this bug. This will occur if the host was originally installed with a level that pre-dates 19.04 (Disco Dingo).

[0] https://lists.ubuntu.com/archives/ubuntu-devel-announce/2018-November/001253.html
[1] https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

Just as information and it was reported by some customers I tested the scenario:

[Test Plan]
Installing Ceph using Bionic / Ceph-Octopus
Upgrade from Bionic to Focal
Upgrade from Ceph Octopus to => Pacific => Quincy using UCA(focal-yoga) repos.
After that the error described in the LP will happen on the clients and ceph-osd, the suggested workaround was applied and the issue was resolved.

This issue needs to be fixed in the UCA repositories as well.

the kinetic version is stuck because of FTBFS

Hello Dan, or anyone else affected,

Accepted ceph into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/17.2.0-0ubuntu0.22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

This bug affects us, and the proposed workaround setting the osd_class_dir worked.

Installed our system at bionic-ussuri (UCA), upgraded to focal, then continued all the way to focal-yoga (UCA). Ceph packages from UCA are currently version 17.2.0-0ubuntu0.22.04.1~cloud0.

Applying the osd_class_dir workaround and restarting ceph-mgr.target on the ceph-mons also got rid of this problem that appeared after upgrading to focal-yoga and Ceph Quincy:

$ sudo ceph status
  cluster:
    id: cb562c8a-787a-11ec-a57e-00163e0f587a
    health: HEALTH_ERR
            Module 'devicehealth' has failed: disk I/O error

Is it possible to have the ceph*17.2.0-0ubuntu0.22.04.2 package in UCA for testing?

This bug was fixed in the package ceph - 17.2.0-0ubuntu4

---------------
ceph (17.2.0-0ubuntu4) kinetic; urgency=medium

  * d/p/fix-kinetic-libfmt.patch: Apply fixes for libfmt > 8.
  * d/p/fix-kinetic-misc.patch: Misc fixes for Ubuntu Kinetic.

 -- Luciano Lo Giudice <email address hidden> Thu, 22 Sep 2022 17:00:12 +0100

I have tested the UCA proposed packages:

 dpkg -l |grep ceph
ii ceph 17.2.0-0ubuntu0.22.04.2~cloud0 amd64 distributed storage and file system

It works!

root@juju-b47d9d-ceph-11:/etc/apt/sources.list.d# sudo ceph daemon osd.3 config get osd_class_dir
{
    "osd_class_dir": "/usr/lib/x86_64-linux-gnu/rados-classes"
}

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package ceph - 17.2.0-0ubuntu0.22.04.2

---------------
ceph (17.2.0-0ubuntu0.22.04.2) jammy; urgency=medium

  * d/p/lp1986747-fix-osd-class-dir.patch: Partially revert upstream
    change that breaks classpath loading (LP: #1986747).

 -- Chris MacNaughton <email address hidden> Thu, 01 Sep 2022 16:30:32 +0100

This bug was fixed in the package ceph - 17.2.0-0ubuntu0.22.04.2~cloud0
---------------

 ceph (17.2.0-0ubuntu0.22.04.2~cloud0) focal-yoga; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (17.2.0-0ubuntu0.22.04.2) jammy; urgency=medium
 .
   * d/p/lp1986747-fix-osd-class-dir.patch: Partially revert upstream
     change that breaks classpath loading (LP: #1986747).