OpenStack Compute (nova)

poor handling of download failure from glance in nova-compute

Bug #801412 reported by Jesse Andrews
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Pádraig Brady
OpenStack Compute (nova) 2012.1 "essex"

Bug Description

Hypervisor: qemu/kvm
nova: trunk as of june 23
host os: maverick
type: multi-node install - nova + glance + openstack api

When nova-compute fails to download an image (for instance if you forget to set your glance_api_servers flag) it will create a 0 sized file:

root@356586-nova-compute-n01:/var/lib/nova/instances/_base# ls -al
drwxr-xr-x 2 nova nogroup 4096 2011-06-23 21:54 .
drwxr-xr-x 8 nova root 4096 2011-06-23 21:54 ..
-rw-r--r-- 1 nova nogroup 0 2011-06-23 21:47 00000001

---

Then on further attempts to launch it doesn't re-attempt to download.

Perhaps instead of downloading the file to the final location we should do a 00000001.download then move it upon finish?

Also checking the checksum before we finish the move?

Jesse Andrews (anotherjesse)
Changed in nova:
importance: Undecided → High
Thierry Carrez (ttx)
Changed in nova:
importance: High → Medium
status: New → Confirmed
Revision history for this message
Jesse Andrews (anotherjesse) wrote :

If this is still an issue we should probably add this to essex. When this triggered it would cause all further launch this image to fail.

Pádraig Brady (p-draigbrady)
Changed in nova:
assignee: nobody → Pádraig Brady (p-draigbrady)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/3664

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/3664
Committed: http://github.com/openstack/nova/commit/3db5b1e7099e1c10d557577350dc28e323c70af4
Submitter: Jenkins
Branch: master

commit 3db5b1e7099e1c10d557577350dc28e323c70af4
Author: Pádraig Brady <email address hidden>
Date: Thu Feb 2 14:56:54 2012 +0000

    fix stale libvirt images on download failure. Bug 801412

    If we've just truncated/created the file to receive
    the download, but the download fails, then delete the file.
    Otherwise subsequent downloads would not be attempted.

    Change-Id: Ibe9ba199b2f424d570ce7aedfb768ce283816a3b

Changed in nova:
status: In Progress → Fix Committed
Brian Waldon (bcwaldon)
Changed in nova:
milestone: none → essex-4
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/4141

Changed in nova:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/4141
Committed: http://github.com/openstack/nova/commit/46f7adadaf1d75f36a4544cb9c011056fd6d0e3a
Submitter: Jenkins
Branch: master

commit 46f7adadaf1d75f36a4544cb9c011056fd6d0e3a
Author: Pádraig Brady <email address hidden>
Date: Tue Feb 14 16:47:00 2012 +0000

    improve stale libvirt images handling fix. Bug 801412

    If the unlink fails (because the path wasn't
    even created for example), then that will shroud
    the original failure. Instead log failure to
    remove the stale image, and raise the original exception.

    Change-Id: I36c6968823bcf81d704319739b3a992dae75266a

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.