MAAS

installation on several nodes failed with errors relating to dmsetup remove of ceph devices.

Bug #1768893 reported by Jason Hobbs
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Undecided
Unassigned
MAAS 2.4.0rc2
curtin
Fix Released
High
Ryan Harper

Bug Description

During a build of FCB, installation on several nodes failed with errors relating to dmsetup remove of ceph devices. Here's an example:

http://paste.ubuntu.com/p/prBvWtQyJC/

versions:

maas 2.3.2-6485-ge93e044-0ubuntu1~16.04.1
python3-curtin_18.1-1-g45564eef-0ubuntu1~16.04.1
Cloud-init v. 18.2

logs: http://people.canonical.com/~jhobbs/infra-logs-19af7264-6ae3-40fc-8bfa-ba90365f2e79.tar

See original description

Related branches

~raharper/curtin:ubuntu/xenial/sru-20180518
Server Team CI bot: Approve (continuous-integration)
curtin developers: Pending requested
Diff: 3718 lines (+1736/-596)
48 files modified
curtin/block/__init__.py (+18/-14)
curtin/block/bcache.py (+87/-0)
curtin/block/clear_holders.py (+90/-52)
curtin/block/iscsi.py (+7/-8)
curtin/block/mdadm.py (+68/-4)
curtin/block/zfs.py (+26/-1)
curtin/commands/apt_config.py (+5/-0)
curtin/commands/block_meta.py (+173/-76)
curtin/commands/curthooks.py (+3/-3)
curtin/commands/install.py (+22/-23)
curtin/util.py (+35/-25)
debian/changelog (+22/-0)
dev/null (+0/-128)
doc/topics/integration-testing.rst (+16/-0)
doc/topics/storage.rst (+61/-4)
examples/tests/dirty_disks_config.yaml (+6/-0)
examples/tests/filesystem_battery.yaml (+23/-0)
examples/tests/lvm.yaml (+21/-0)
tests/unittests/helpers.py (+3/-1)
tests/unittests/test_block_zfs.py (+96/-0)
tests/unittests/test_clear_holders.py (+87/-38)
tests/unittests/test_commands_block_meta.py (+425/-25)
tests/unittests/test_commands_install.py (+28/-0)
tests/unittests/test_make_dname.py (+28/-1)
tests/unittests/test_util.py (+47/-0)
tests/vmtests/__init__.py (+146/-19)
tests/vmtests/helpers.py (+49/-32)
tests/vmtests/test_basic.py (+12/-9)
tests/vmtests/test_centos_basic.py (+0/-2)
tests/vmtests/test_fs_battery.py (+49/-0)
tests/vmtests/test_lvm.py (+6/-10)
tests/vmtests/test_lvm_iscsi.py (+8/-2)
tests/vmtests/test_mdadm_bcache.py (+7/-73)
tests/vmtests/test_network.py (+0/-1)
tests/vmtests/test_network_alias.py (+0/-1)
tests/vmtests/test_network_bonding.py (+0/-1)
tests/vmtests/test_network_bridging.py (+0/-1)
tests/vmtests/test_network_ipv6.py (+0/-1)
tests/vmtests/test_network_mtu.py (+0/-1)
tests/vmtests/test_network_static.py (+0/-1)
tests/vmtests/test_network_vlan.py (+6/-1)
tests/vmtests/test_nvme.py (+0/-18)
tests/vmtests/test_pollinate_useragent.py (+2/-2)
tests/vmtests/test_raid5_bcache.py (+0/-4)
tests/vmtests/test_uefi_basic.py (+0/-9)
tests/vmtests/test_zfsroot.py (+40/-1)
tools/jenkins-runner (+12/-0)
tools/vmtest-sync-images (+2/-4)
~raharper/curtin:ubuntu/artful/sru-20180518
Server Team CI bot: Approve (continuous-integration)
curtin developers: Pending requested
Diff: 3718 lines (+1736/-596)
48 files modified
curtin/block/__init__.py (+18/-14)
curtin/block/bcache.py (+87/-0)
curtin/block/clear_holders.py (+90/-52)
curtin/block/iscsi.py (+7/-8)
curtin/block/mdadm.py (+68/-4)
curtin/block/zfs.py (+26/-1)
curtin/commands/apt_config.py (+5/-0)
curtin/commands/block_meta.py (+173/-76)
curtin/commands/curthooks.py (+3/-3)
curtin/commands/install.py (+22/-23)
curtin/util.py (+35/-25)
debian/changelog (+22/-0)
dev/null (+0/-128)
doc/topics/integration-testing.rst (+16/-0)
doc/topics/storage.rst (+61/-4)
examples/tests/dirty_disks_config.yaml (+6/-0)
examples/tests/filesystem_battery.yaml (+23/-0)
examples/tests/lvm.yaml (+21/-0)
tests/unittests/helpers.py (+3/-1)
tests/unittests/test_block_zfs.py (+96/-0)
tests/unittests/test_clear_holders.py (+87/-38)
tests/unittests/test_commands_block_meta.py (+425/-25)
tests/unittests/test_commands_install.py (+28/-0)
tests/unittests/test_make_dname.py (+28/-1)
tests/unittests/test_util.py (+47/-0)
tests/vmtests/__init__.py (+146/-19)
tests/vmtests/helpers.py (+49/-32)
tests/vmtests/test_basic.py (+12/-9)
tests/vmtests/test_centos_basic.py (+0/-2)
tests/vmtests/test_fs_battery.py (+49/-0)
tests/vmtests/test_lvm.py (+6/-10)
tests/vmtests/test_lvm_iscsi.py (+8/-2)
tests/vmtests/test_mdadm_bcache.py (+7/-73)
tests/vmtests/test_network.py (+0/-1)
tests/vmtests/test_network_alias.py (+0/-1)
tests/vmtests/test_network_bonding.py (+0/-1)
tests/vmtests/test_network_bridging.py (+0/-1)
tests/vmtests/test_network_ipv6.py (+0/-1)
tests/vmtests/test_network_mtu.py (+0/-1)
tests/vmtests/test_network_static.py (+0/-1)
tests/vmtests/test_network_vlan.py (+6/-1)
tests/vmtests/test_nvme.py (+0/-18)
tests/vmtests/test_pollinate_useragent.py (+2/-2)
tests/vmtests/test_raid5_bcache.py (+0/-4)
tests/vmtests/test_uefi_basic.py (+0/-9)
tests/vmtests/test_zfsroot.py (+40/-1)
tools/jenkins-runner (+12/-0)
tools/vmtest-sync-images (+2/-4)
~raharper/curtin:ubuntu/bionic/sru-20180518
Server Team CI bot: Approve (continuous-integration)
curtin developers: Pending requested
Diff: 3486 lines (+1640/-558)
44 files modified
curtin/block/__init__.py (+18/-14)
curtin/block/bcache.py (+87/-0)
curtin/block/clear_holders.py (+88/-51)
curtin/block/iscsi.py (+7/-8)
curtin/block/mdadm.py (+65/-0)
curtin/block/zfs.py (+26/-1)
curtin/commands/apt_config.py (+5/-0)
curtin/commands/block_meta.py (+173/-76)
curtin/commands/curthooks.py (+2/-2)
curtin/util.py (+35/-25)
debian/changelog (+18/-0)
dev/null (+0/-128)
doc/topics/integration-testing.rst (+16/-0)
doc/topics/storage.rst (+61/-4)
examples/tests/dirty_disks_config.yaml (+6/-0)
examples/tests/filesystem_battery.yaml (+23/-0)
tests/unittests/helpers.py (+3/-1)
tests/unittests/test_block_zfs.py (+96/-0)
tests/unittests/test_clear_holders.py (+82/-33)
tests/unittests/test_commands_block_meta.py (+425/-25)
tests/unittests/test_make_dname.py (+28/-1)
tests/unittests/test_util.py (+47/-0)
tests/vmtests/__init__.py (+145/-18)
tests/vmtests/helpers.py (+49/-32)
tests/vmtests/test_basic.py (+12/-9)
tests/vmtests/test_centos_basic.py (+0/-2)
tests/vmtests/test_fs_battery.py (+49/-0)
tests/vmtests/test_lvm.py (+5/-9)
tests/vmtests/test_lvm_iscsi.py (+8/-2)
tests/vmtests/test_mdadm_bcache.py (+7/-73)
tests/vmtests/test_network.py (+0/-1)
tests/vmtests/test_network_alias.py (+0/-1)
tests/vmtests/test_network_bonding.py (+0/-1)
tests/vmtests/test_network_bridging.py (+0/-1)
tests/vmtests/test_network_ipv6.py (+0/-1)
tests/vmtests/test_network_mtu.py (+0/-1)
tests/vmtests/test_network_static.py (+0/-1)
tests/vmtests/test_network_vlan.py (+0/-1)
tests/vmtests/test_nvme.py (+0/-18)
tests/vmtests/test_raid5_bcache.py (+0/-4)
tests/vmtests/test_uefi_basic.py (+0/-9)
tests/vmtests/test_zfsroot.py (+40/-1)
tools/jenkins-runner (+12/-0)
tools/vmtest-sync-images (+2/-4)
~raharper/curtin:fix/clear-holders-refactor
Scott Moser (community): Approve
Server Team CI bot: Approve (continuous-integration)
Diff: 679 lines (+250/-96)
7 files modified
curtin/block/__init__.py (+18/-14)
curtin/block/clear_holders.py (+86/-54)
curtin/block/iscsi.py (+7/-8)
curtin/block/mdadm.py (+65/-0)
tests/unittests/test_clear_holders.py (+66/-20)
tests/vmtests/test_lvm.py (+5/-0)
tests/vmtests/test_lvm_iscsi.py (+3/-0)
Jason Hobbs (jason-hobbs)
tags: added: cdo-qa cdo-qa-blocker foundations-engine
description: updated
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Jason,

Could you please attach the Curtin configuration sent by MAAS: maas <user> machine get-curtin-config <system_id>

That said, it seems that the 'dmsetup' is what's failing, it could be a proper error from dmsetup or a potential issue with curtin ?

Changed in maas:
milestone: none → 2.4.0rc1
status: New → Incomplete
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

Andres - there is no good way to get that during automated tests (see bug 1768911). Shouldn't maas/curtin be logging that if it's required to debug issues?

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

Also, since the tarball includes a database dump, shouldn't you have all the information you need to see what the curtin config would be, since it's generated from information in the database?

Changed in maas:
status: Incomplete → New
Revision history for this message
Blake Rouse (blake-rouse) wrote :

Jason,

You can use get-curtin-config even when the machine has failed deployment. Correct about the database dump, we could load the database and make the same command.

Doing that in QA anytime you get a failed deployment would be better.

Ryan Harper (raharper)
Changed in curtin:
importance: Undecided → High
status: New → Incomplete
Andres Rodriguez (andreserl)
Changed in maas:
status: New → Incomplete
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Marking this as incomplete. It doesn't seem like a MAAS issue at first glance.

The yaml config is required by curtin developers to determine what the storage configuration is.

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

15:17 < ivoks> charm [ceph-osd] creates vgs and lvs for ceph
15:17 < ivoks> and curtin fails to clean those up properly
15:17 < jhobbs> that's new right? the vgs and lvs
15:17 < ivoks> yes, that's in -next charms

Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.4.0rc1 → 2.4.0rc2
David Britton (dpb)
Changed in curtin:
status: Incomplete → In Progress
assignee: nobody → Ryan Harper (raharper)
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Marking this as invalid for MAAS as this is an issue in curtin which is already being addressed.

Changed in maas:
status: Incomplete → Invalid
Revision history for this message
Scott Moser (smoser) wrote :

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=4bf7750b

Changed in curtin:
status: In Progress → Fix Committed
Revision history for this message
Ryan Harper (raharper) wrote : Fixed in curtin version 18.1-17-gae48e86f-0ubuntu1.

This bug is believed to be fixed in curtin in version18.1-17-gae48e86f-0ubuntu1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.