Comment 3 for bug 1986747

While attempting to recreate this issue, we noticed a new Jammy/Quincy installation will load the correct library files!

The freshly deployed Quincy reports the incorrect `osd_class_dir` (lib/x86_64-linux-gnu/rados-classes), but lsof shows the ceph-osd process is loading rados-classes libraries from /usr/lib:
ceph-osd 735836 64045 mem REG 8,2 530688 35521076 /usr/lib/x86_64-linux-gnu/rados-classes/libcls_rbd.so.1.0.0

How is this happening?

It turns out that the correct library file is loaded by accident. New Jammy installations have /lib symlinked to /usr/lib. The ceph-osd processes have a working directory of "/" so the relative "lib/" path follows the symlink to the correct "/usr/lib" location.

Starting with 19.04, merged-usr is now the default for new installations [0]. This merged-usr feature [1] is what creates the symlink from /lib -> /usr/lib.

The key thing to note here is that these symlinks do not get created on hosts that have been upgraded:
"Existing systems, upon upgrade, will not be reconfigured for merged-usr."

Hosts without the /lib -> /usr/lib symlink are exposed to the issue described in this bug. This will occur if the host was originally installed with a level that pre-dates 19.04 (Disco Dingo).

[0] https://lists.ubuntu.com/archives/ubuntu-devel-announce/2018-November/001253.html
[1] https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/