Glance will leak staging data

Bug #1913625 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Undecided
Dan Smith

Bug Description

In various situations, glance will leak (potentially very large) temporary files in the staging store.

One example is doing a web-download import, where glance initially downloads the image to its staging store. If the worker doing that activity crashes, loses power, etc, the user may delete the image and try again on another worker. When the crashed worker resumes, the staging data will remain but nothing will ever clean it up.

Another example would be a misconfigured glance that uses local staging directories, but glance-direct is used, where the user stages data, and then deletes the image from another worker.

Even in a situation where shared staging is properly configured, a failure to access the staging location during the delete call will result in the image being deleted, but the staging file not being purged.

IMHO, glance workers should clean their staging directories at startup, purging any data that is attributable to a previous image having been deleted.

Another option is to add a store location for each staged image, and make sure the scrubber can clean those things from the staging directory periodically (this requires also running the scrubber on each node, which may not be common practice currently).

Revision history for this message
Dan Smith (danms) wrote :

Marking this as invalid as the glance team thinks this is not a bug.

Changed in glance:
status: New → Invalid
Dan Smith (danms)
Changed in glance:
status: Invalid → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 21.0.0.0b3

This issue was fixed in the openstack/glance 21.0.0.0b3 development milestone.

Changed in glance:
status: Confirmed → Fix Released
assignee: nobody → Dan Smith (danms)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to glance (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/glance/+/785551
Committed: https://opendev.org/openstack/glance/commit/54060e8b72b048d9a6a2ff2ca01a5fc18b6eafec
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 54060e8b72b048d9a6a2ff2ca01a5fc18b6eafec
Author: Dan Smith <email address hidden>
Date: Tue Feb 23 18:59:07 2021 -0800

    Make functional tests set node_staging_uri

    Currently it is not possible to configure the staging directory
    URI of the functional workers. We need to be able to do that in order
    to enable the stage cleaning behavior. Right now, they're all sharing
    /tmp/staging, which will cause workers to conflict while running in
    parallel. This causes them to use their private test directory,
    which may also help some other spurious failures due to interaction.

    Related-Bug: #1913625
    Change-Id: Ic2ac2a528206c50c38e948a096daf9eb8e5eb715
    (cherry picked from commit e9852fb6251c655f0e384c6b0b8677bd556f17a1)
    (cherry picked from commit 8e216b4913e8b842cab983c100165949bed7def2)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/glance/+/785552
Committed: https://opendev.org/openstack/glance/commit/e87d3fbbe5e0bd404337a0e82b496edc30370e63
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit e87d3fbbe5e0bd404337a0e82b496edc30370e63
Author: Dan Smith <email address hidden>
Date: Mon Feb 22 12:52:03 2021 -0800

    Add housekeeping module and staging cleaner

    As noted in previous discussions, glance should clean its staging
    directory on startup. This is important for scenarios where we
    started an import operation, but failed in the middle. If, when we
    recover, the image has already been deleted from the database, then
    we will never remove the (potentially very large) residue from disk
    in our staging directory.

    This is currently a problem with web-download, but will also occur
    with glance-direct once we have the non-shared distributed import
    functionality merged.

    Conflicts:
            glance/tests/unit/common/test_wsgi_app.py: Does not exist

    Closes-Bug: #1913625
    Change-Id: Ib80e9cfb58680f9e8ead5993dc206f4da882dd09
    (cherry picked from commit 232177e68c547a815c31a2f30b6d1f95cdb3098d)
    (cherry picked from commit 94679941368d37520975e0a1dc81d27424167191)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 20.1.0

This issue was fixed in the openstack/glance 20.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 21.1.0

This issue was fixed in the openstack/glance 21.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.