Failed import of one store will remain in progress forever if all_stores_must_succeed=True

Bug #1891352 reported by Dan Smith
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Undecided
Dan Smith
Ussuri
Fix Committed
Undecided
Dan Smith

Bug Description

If import is called with all_stores_must_succeed=True and a store fails during set_image_data(), the store will remain in os_glance_importing_stores forever, never going into the os_glance_failed_import list. This means a polling client will never notice that the import failed. Further, if multiple stores are included in the import, and the failure happens in the later stores, the revert process will remove the earlier stores (after they had already been reported as available in stores). This means a polling client doing an import on an image already in store1 to store2,store3,store4 will see the following progression:

stores=store1;os_glance_importing_to_stores=store2,store3,store4

stores=store1,store2;os_glance_importing_to_stores=store3,store4

stores=store1,store2,store3;os_glance_importing_to_stores=store4

stores=store1,store2;os_glance_importing_to_stores=store4

stores=store1;os_glance_importing_to_stores=store4

The last line, forever, and never see anything in os_glance_failed_import

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.opendev.org/745932
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=ec372377b1e69eb143e4d08b0041b8199459ee15
Submitter: Zuul
Branch: master

commit ec372377b1e69eb143e4d08b0041b8199459ee15
Author: Dan Smith <email address hidden>
Date: Wed Aug 12 09:28:46 2020 -0700

    Functional reproducer for bug 1891352

    This reproduces the behavior described in the referenced bug, showing
    the behavior before the fix. This asserts the *broken* behavior and leaves
    examples of what the asserts should be for the fix patch.

    Partial-Bug: #1891352

    Change-Id: I992a14abb7cb41e2472741bf08b93976d4320ace

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/745933
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=737dfca83c8d347ef22eab7afca4057ce5f03f20
Submitter: Zuul
Branch: master

commit 737dfca83c8d347ef22eab7afca4057ce5f03f20
Author: Dan Smith <email address hidden>
Date: Wed Aug 12 09:22:46 2020 -0700

    Fix import failure status reporting when all_stores_must_succeed=True

    As described in the referenced bug, if a store fails with
    all_stores_must_succeed=True, we never add that store to the
    failed list, nor remove it from the pending-import list, leaving a
    polling client to wait forever. This fixes that by making the revert
    handler of the import task do that if we are the one that failed.

    Closes-Bug: #1891352
    Change-Id: I3571960bbfb4f8f0a716937b541f5b1594cd0e16

Changed in glance:
status: In Progress → Fix Released
Changed in glance:
assignee: nobody → Dan Smith (danms)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/748011

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/748012

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/ussuri)

Reviewed: https://review.opendev.org/748011
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=931d39ec382a187ca52babe4c3d2d3516b079465
Submitter: Zuul
Branch: stable/ussuri

commit 931d39ec382a187ca52babe4c3d2d3516b079465
Author: Dan Smith <email address hidden>
Date: Wed Aug 12 09:28:46 2020 -0700

    Functional reproducer for bug 1891352

    This reproduces the behavior described in the referenced bug, showing
    the behavior before the fix. This asserts the *broken* behavior and leaves
    examples of what the asserts should be for the fix patch.

    Partial-Bug: #1891352

    Change-Id: I992a14abb7cb41e2472741bf08b93976d4320ace
    (cherry picked from commit ec372377b1e69eb143e4d08b0041b8199459ee15)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/748012
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=2d5e84390e1a9960abc6cd2969ee7021c62c4c33
Submitter: Zuul
Branch: stable/ussuri

commit 2d5e84390e1a9960abc6cd2969ee7021c62c4c33
Author: Dan Smith <email address hidden>
Date: Wed Aug 12 09:22:46 2020 -0700

    Fix import failure status reporting when all_stores_must_succeed=True

    As described in the referenced bug, if a store fails with
    all_stores_must_succeed=True, we never add that store to the
    failed list, nor remove it from the pending-import list, leaving a
    polling client to wait forever. This fixes that by making the revert
    handler of the import task do that if we are the one that failed.

    Closes-Bug: #1891352
    Change-Id: I3571960bbfb4f8f0a716937b541f5b1594cd0e16
    (cherry picked from commit 737dfca83c8d347ef22eab7afca4057ce5f03f20)

Revision history for this message
prabha (shrudhavishal) wrote :

Swift backend to my Glance service has the same error as OS_Glance_Image_importing failed.But as I refer the bug as previously mentioned ,I dont see the term all_stores_must_succeed in glance-api.conf file
I am using Ussuri version
Please help me to solve this

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 20.1.0

This issue was fixed in the openstack/glance 20.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.