/dev/sdb got mounted root, 'guess' algorithm exludes /dev/sda (among others)

Bug #1350049 reported by David Britton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
swift-storage (Juju Charms Collection)
Fix Released
Low
Chris Glass

Bug Description

Full unit log: http://paste.ubuntu.com/7898450/

for some reason on a juju (1.20.2 prerelease) + maas (1.6.0+bzr2539-0ubuntu1~beta6~ppa1) deploy, I got /dev/sdb as my root device. swift-storage didn't like this.

/dev/sda was free and could have been used, but the charm didn't like this in it's 'guess' algorithm.

ubuntu@bohr:~$ tail -30 /var/log/juju/unit-swift-storage-0.log
2014-07-29 20:29:34 INFO install GPT data structures destroyed! You may now partition the disk using fdisk or
2014-07-29 20:29:34 INFO install other utilities.
2014-07-29 20:29:34 INFO install Warning: The kernel is still using the old partition table.
2014-07-29 20:29:34 INFO install The new table will be used at the next reboot.
2014-07-29 20:29:34 INFO install The operation has completed successfully.
2014-07-29 20:29:34 INFO install 1+0 records in
2014-07-29 20:29:34 INFO install 1+0 records out
2014-07-29 20:29:34 INFO install 1048576 bytes (1.0 MB) copied, 0.00211706 s, 495 MB/s
2014-07-29 20:29:34 INFO install 100+0 records in
2014-07-29 20:29:34 INFO install 100+0 records out
2014-07-29 20:29:34 INFO install 51200 bytes (51 kB) copied, 0.00456763 s, 11.2 MB/s
2014-07-29 20:29:34 INFO install mkfs.xfs: cannot open /dev/sdb: Device or resource busy
2014-07-29 20:29:34 INFO install <open file '/proc/partitions', mode 'r' at 0x7ffedc45fdb0>
2014-07-29 20:29:34 INFO install Traceback (most recent call last):
2014-07-29 20:29:34 INFO install File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/install", line 91, in <module>
2014-07-29 20:29:34 INFO install main()
2014-07-29 20:29:34 INFO install File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/install", line 85, in main
2014-07-29 20:29:34 INFO install hooks.execute(sys.argv)
2014-07-29 20:29:34 INFO install File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/charmhelpers/core/hookenv.py", line 478, in execute
2014-07-29 20:29:34 INFO install self._hooks[hook_name]()
2014-07-29 20:29:34 INFO install File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/install", line 45, in install
2014-07-29 20:29:34 INFO install setup_storage()
2014-07-29 20:29:34 INFO install File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift_storage_utils.py", line 180, in setup_storage
2014-07-29 20:29:34 INFO install mkfs_xfs(dev)
2014-07-29 20:29:34 INFO install File "/var/lib/juju/agents/unit-swift-storage-0/charm/hooks/swift_storage_utils.py", line 172, in mkfs_xfs
2014-07-29 20:29:34 INFO install check_call(cmd)
2014-07-29 20:29:34 INFO install File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
2014-07-29 20:29:34 INFO install raise CalledProcessError(retcode, cmd)
2014-07-29 20:29:34 INFO install subprocess.CalledProcessError: Command '['mkfs.xfs', '-f', '-i', 'size=1024', '/dev/sdb']' returned non-zero exit status 1
2014-07-29 20:29:34 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1
ubuntu@bohr:~$ ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Jul 29 20:27 /dev/sda
brw-rw---- 1 root disk 8, 1 Jul 29 20:27 /dev/sda1
brw-rw---- 1 root disk 8, 2 Jul 29 20:27 /dev/sda2
brw-rw---- 1 root disk 8, 16 Jul 29 20:29 /dev/sdb
brw-rw---- 1 root disk 8, 17 Jul 29 20:27 /dev/sdb1
ubuntu@bohr:~$ ll /proc/partitions
-r--r--r-- 1 root root 0 Jul 29 20:34 /proc/partitions
ubuntu@bohr:~$ cat /proc/partitions
major minor #blocks name

   8 0 119454720 sda
   8 1 118405103 sda1
   8 2 1047552 sda2
   8 16 117220824 sdb
   8 17 117219800 sdb1

# /dev/sdb is root (!)
# The landscape cloud-installer tells the charm to use sd[a-z] more or less.

ubuntu@bohr:~$ mount |grep sdb
/dev/sdb1 on / type ext4 (rw)
ubuntu@bohr:~$ mount |grep sda
ubuntu@bohr:~$ ll /dev/sda*
brw-rw---- 1 root disk 8, 0 Jul 29 20:27 /dev/sda
brw-rw---- 1 root disk 8, 1 Jul 29 20:27 /dev/sda1
brw-rw---- 1 root disk 8, 2 Jul 29 20:27 /dev/sda2

Related branches

Revision history for this message
James Page (james-page) wrote :

The charm makes the assumption that the first disk is off limits; I guess its possible to look at devices and see if they are in use and exclude that way instead - ceph does something like this.

Changed in swift-storage (Juju Charms Collection):
importance: Undecided → Low
status: New → Triaged
Chris Glass (tribaal)
Changed in swift-storage (Juju Charms Collection):
assignee: nobody → Chris Glass (tribaal)
status: Triaged → In Progress
Revision history for this message
Chris Glass (tribaal) wrote :

A fix for theis was committed - I changed the logic to test if the partition/device is mounted instead of relying on a blacklist.
It is released to the charm store as of cs:trusty/swift-storage-3

Changed in swift-storage (Juju Charms Collection):
status: In Progress → Fix Committed
status: Fix Committed → Fix Released
Revision history for this message
Björn Tillenius (bjornt) wrote :

It looks like this got fixed in the stable branches only. I ran into this using the /next charms, and looking at the code, the blacklist is still there.

Revision history for this message
Björn Tillenius (bjornt) wrote :

I filed bug 1379390 for forward-port this fix to the /next branches.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.