OpenStack Compute (nova)

file injection fails when using fallback method

Bug #1484586 reported by Paul Carlton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Alexis Lee

Bug Description

Trying to perform file injection without libguestfs, i.e. fallback to using nbd.

2015-08-13 13:21:21 43295 DEBUG oslo_concurrency.processutils [req-e70d20d6-f42c-4d3e-8598-4dd1a76e09f5 61e0c3d7a0204bdeb647498c47599bc0 c942c664c4024ce4b5fe2bf8c3a21a3c] Running cmd (subprocess): sudo nova-rootwrap /opt/stack/service/nova-compute/etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd8 //var/lib/nova/instances/e8cb4369-adf8-4e97-ad75-9d181d3c9dac/disk execute /opt/stack/venv/nova-20150805T111039Z/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
2015-08-13 13:21:21 43295 DEBUG oslo_concurrency.processutils [req-e70d20d6-f42c-4d3e-8598-4dd1a76e09f5 61e0c3d7a0204bdeb647498c47599bc0 c942c664c4024ce4b5fe2bf8c3a21a3c] CMD "sudo nova-rootwrap /opt/stack/service/nova-compute/etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd8 //var/lib/nova/instances/e8cb4369-adf8-4e97-ad75-9d181d3c9dac/disk" returned: 0 in 0.096s execute /opt/stack/venv/nova-20150805T111039Z/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
2015-08-13 13:21:21 43295 DEBUG oslo_concurrency.lockutils [req-e70d20d6-f42c-4d3e-8598-4dd1a76e09f5 61e0c3d7a0204bdeb647498c47599bc0 c942c664c4024ce4b5fe2bf8c3a21a3c] Lock "nbd-allocation-lock" released by "_inner_get_dev" :: held 0.099s inner /opt/stack/venv/nova-20150805T111039Z/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456
2015-08-13 13:21:21 43295 DEBUG nova.virt.disk.mount.api [req-e70d20d6-f42c-4d3e-8598-4dd1a76e09f5 61e0c3d7a0204bdeb647498c47599bc0 c942c664c4024ce4b5fe2bf8c3a21a3c] Map dev /dev/nbd8 map_dev /opt/stack/venv/nova-20150805T111039Z/lib/python2.7/site-packages/nova/virt/disk/mount/api.py:140
2015-08-13 13:21:21 43295 DEBUG oslo_concurrency.processutils [req-e70d20d6-f42c-4d3e-8598-4dd1a76e09f5 61e0c3d7a0204bdeb647498c47599bc0 c942c664c4024ce4b5fe2bf8c3a21a3c] Running cmd (subprocess): sudo nova-rootwrap /opt/stack/service/nova-compute/etc/nova/rootwrap.conf kpartx -a /dev/nbd8 execute /opt/stack/venv/nova-20150805T111039Z/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
2015-08-13 13:21:21 43295 DEBUG oslo_concurrency.processutils [req-e70d20d6-f42c-4d3e-8598-4dd1a76e09f5 61e0c3d7a0204bdeb647498c47599bc0 c942c664c4024ce4b5fe2bf8c3a21a3c] CMD "sudo nova-rootwrap /opt/stack/service/nova-compute/etc/nova/rootwrap.conf kpartx -a /dev/nbd8" returned: 0 in 0.093s execute /opt/stack/venv/nova-20150805T111039Z/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225

2015-08-13 13:21:21 43295 DEBUG nova.virt.disk.mount.api [req-e70d20d6-f42c-4d3e-8598-4dd1a76e09f5 61e0c3d7a0204bdeb647498c47599bc0 c942c664c4024ce4b5fe2bf8c3a21a3c] Fail to mount, tearing back down do_mount /opt/stack/venv/nova-20150805T111039Z/lib/python2.7/site-packages/nova/virt/disk/mount/api.py:223

Although the kpartx command works the check for file path fails generating an error.

Inserting a short sleep before checking for the path seems to work. This issue is obviously timing related and I do not encounter this when running devstack on a libvirt host. However it occurs on some of the baremetal hypervisors in our lab very reliably.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/212619

Changed in nova:
assignee: nobody → Paul Carlton (paul-carlton2)
status: New → In Progress
Revision history for this message
Paul Carlton (paul-carlton2) wrote :

https://review.openstack.org/#/c/212619/ to fix this

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Paul Carlton (<email address hidden>) on branch: master
Review: https://review.openstack.org/212619
Reason: Told I need to submit test for this function before it can be changed

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/287442

Changed in nova:
assignee: Paul Carlton (paul-carlton2) → Lee Yarwood (lyarwood)
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Lee Yarwood (lyarwood) → Alexis Lee (alexisl)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/215613
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2c1b19761b3d960055ced11558dda22d022d77f4
Submitter: Jenkins
Branch: master

commit 2c1b19761b3d960055ced11558dda22d022d77f4
Author: Alexis Lee <email address hidden>
Date: Fri Aug 21 13:58:06 2015 +0100

    Wait for device to be mapped

    There's a race condition when trying to perform file injection without
    libguestfs, which causes a fallback to nbd device. Although the kpartx
    command succeeds, it does so after the code has tested for success, so
    Nova thinks it failed.

    Retry a few times to avoid this.

    Co-Authored-By: Paul Carlton <email address hidden>
    Change-Id: Ie5c186562475cd56c55520ad7123f47a0130b2a4
    Closes-Bug: #1428639
    Closes-Bug: #1484586

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Lee Yarwood (<email address hidden>) on branch: master
Review: https://review.openstack.org/287442
Reason: The change originally proposed by Alexis has merged.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 14.0.0.0b1

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.