Can not mount cephfs in VM from cloud image

Bug #1164739 reported by Stephen Street
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Stefan Bader
Precise
Fix Released
Medium
Stefan Bader
Quantal
Fix Released
Medium
Stefan Bader

Bug Description

SRU Justification:

Impact: The ceph module is not loadable in a default virtual installation because it depends on the libceph module.

Fix: Add libceph.ko to the appropriate lists (packaging change only).

Testcase: Having the virtual flavour (or meta package) installed only (in Precise a separate flavour still, in Quantal this just does not include the extras package), modprobe for ceph will fail without that change.

---

Install cloud-image based VM. Install cephfs tools

# Install packages
apt-get update
apt-get dist-upgrade
reboot
apt-get install -y ceph-common ceph-fs-common

# Mount ceph filesystem
mount -t ceph 192.168.3.3:/ /mnt
FATAL: Error inserting ceph (/lib/modules/3.5.0-26-generic/kernel/fs/ceph/ceph.ko): Unknown symbol in module, or unknown parameter (see dmesg)
mount.ceph: modprobe failed, exit status 1
mount error: ceph filesystem not supported by the system

# dmesg reports:
[ 251.494403] ceph: Unknown symbol ceph_con_keepalive (err 0)
[ 251.494443] ceph: Unknown symbol ceph_create_client (err 0)
[ 251.494456] ceph: Unknown symbol ceph_calc_pg_primary (err 0)
[ 251.494459] ceph: Unknown symbol ceph_osdc_release_request (err 0)
[ 251.494471] ceph: Unknown symbol ceph_con_open (err 0)
....

Release:

lsb_release -rd
Description: Ubuntu 12.10
Release: 12.10

But also present in 13.04

Related bugs:
 * bug 1063784: Ceph module not installed

---
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Apr 4 22:14 seq
 crw-rw---T 1 root audio 116, 33 Apr 4 22:14 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.6.1-0ubuntu10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 12.10
Ec2AMI: ami-00000005
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: cloud-zone
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Bochs Bochs
MarkForUpload: True
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.5.0-26-generic root=UUID=9e1c8c9f-1a83-4cfd-8774-5d824b8bf616 ro console=tty1 console=ttyS0
ProcVersionSignature: User Name 3.5.0-26.42-generic 3.5.7.6
RelatedPackageVersions:
 linux-restricted-modules-3.5.0-26-generic N/A
 linux-backports-modules-3.5.0-26-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: quantal ec2-images
Uname: Linux 3.5.0-26-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2007:svnBochs:pnBochs:pvr:cvnBochs:ct1:cvr:
dmi.product.name: Bochs
dmi.sys.vendor: Bochs

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote :

Root cause is the linux-image-virtual does not install the kernel module kernel/net/ceph/libceph.ko

find /lib/modules/3.5.0-26-generic -type f -name "*ceph*" returns

/lib/modules/3.5.0-26-generic/kernel/fs/ceph/ceph.ko

Work around by

apt-get install linux-image-extra-virtual

find /lib/modules/3.5.0-26-generic -type f -name "*ceph*" returns

/lib/modules/3.5.0-26-generic/kernel/fs/ceph/ceph.ko
/lib/modules/3.5.0-26-generic/kernel/net/ceph/libceph.ko

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote :

IMHO (really humble), it seems a little strange that kernel/fs/ceph/ceph.so is included in the cloud images but not kernel/net/ceph/libceph.so, its required dependency. Most likely, ceph.so needs to moved to the linux-image-extra-virtual package so that a partial install of the ceph kernel functionality does not happen.

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1164739

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: quantal
Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : BootDmesg.txt

apport information

tags: added: apport-collected ec2-images
description: updated
Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : Lspci.txt

apport information

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : ProcModules.txt

apport information

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : UdevDb.txt

apport information

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : UdevLog.txt

apport information

Revision history for this message
Stephen Street (stephen-redrocketcomputing) wrote : WifiSyslog.txt

apport information

Scott Moser (smoser)
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :

I've chatted some with steven and there are two issues here
1. ceph.ko is included, but it's dependency libceph is not. that seems like a general issue in the package build.
2. mounting ceph filesystems is not supported by Linux-image-virtual.

I don't have strong feelings either way on whether or not ceph should be in -virtual. per Steven, getting it functional would be ~200k. to my knowledge, we don't really have a strong policy on what should be in -virtual other than the general goal of supporting boot on common hypervisor configurations.

Changed in linux (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

I opened bug 1166197 to address the issue of manual modules list not getting automatic dependencies resolved.

Scott Moser (smoser)
description: updated
Revision history for this message
Scott Moser (smoser) wrote :

kernel team, please go ahead and include ceph.ko and libceph.ko so that ceph is usable. We originally decided to add ceph support under bug 1063784.

Revision history for this message
Stefan Bader (smb) wrote :

For reference, ceph was requested to be included by bug #1063784. Seems the missing libceph was not caught back then.

Stefan Bader (smb)
Changed in linux (Ubuntu):
assignee: nobody → Stefan Bader (stefan-bader-canonical)
status: Confirmed → In Progress
Changed in linux (Ubuntu Precise):
importance: Undecided → Medium
Changed in linux (Ubuntu Quantal):
importance: Undecided → Medium
Changed in linux (Ubuntu Precise):
status: New → Triaged
Changed in linux (Ubuntu Quantal):
status: New → Triaged
Stefan Bader (smb)
Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Precise):
assignee: nobody → Stefan Bader (stefan-bader-canonical)
Changed in linux (Ubuntu Quantal):
assignee: nobody → Stefan Bader (stefan-bader-canonical)
status: Triaged → In Progress
Stefan Bader (smb)
description: updated
Stefan Bader (smb)
Changed in linux (Ubuntu Precise):
status: Triaged → In Progress
Tim Gardner (timg-tpi)
Changed in linux (Ubuntu Quantal):
status: In Progress → Fix Committed
Stefan Bader (smb)
Changed in linux (Ubuntu Precise):
status: In Progress → Fix Committed
Revision history for this message
Julian Wiedmann (jwiedmann) wrote :

Landed in 3.8.0-18.28 for raring.

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed' to 'verification-done'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-precise
tags: added: verification-needed-quantal
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Julian Wiedmann (jwiedmann) wrote :

Released with 3.2.0-44.69 and 3.5.0-31.52.

Changed in linux (Ubuntu Quantal):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.