Merge ~raharper/cloud-init:fix/cloud-tests-lxd-retry-delete into cloud-init:master

Proposed by Ryan Harper on 2019-10-11
Status: Merged
Approved by: Dan Watkins on 2019-10-11
Approved revision: aaf0c0e11ed07b65d15f8759d907483561c40334
Merge reported by: Server Team CI bot
Merged at revision: not available
Proposed branch: ~raharper/cloud-init:fix/cloud-tests-lxd-retry-delete
Merge into: cloud-init:master
Diff against target: 32 lines (+13/-1)
1 file modified
tests/cloud_tests/platforms/lxd/instance.py (+13/-1)
Reviewer Review Type Date Requested Status
Dan Watkins 2019-10-11 Approve on 2019-10-11
Server Team CI bot continuous-integration Approve on 2019-10-11
Paride Legovini (community) Approve on 2019-10-11
Review via email: mp+374043@code.launchpad.net

Commit message

cloud_test/lxd: Retry container delete a few times

LXD integration tests fail sometimes due to failure to delete the
container, usually related to ZFS backend. This is a transient
issue unrelated to the test itself. Teach LXD platform to retry
this a few times before returning an error.

To post a comment you must log in.
Paride Legovini (legovini) wrote :

LGTM

review: Approve
Ryan Harper (raharper) wrote :

I need to add a sleep after the failure

FAILED: Continuous integration, rev:c6bdca59c2b7afdf19b9bc7a4df8367584016627
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1206/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1206//rebuild

review: Needs Fixing (continuous-integration)
aaf0c0e... by Ryan Harper on 2019-10-11

Add the sleep before next retry.

Paride Legovini (legovini) wrote :

Right. In the usual failure mode retrying should work even without sleeping, but it's way better to add a sleep(1) there.

review: Approve

PASSED: Continuous integration, rev:aaf0c0e11ed07b65d15f8759d907483561c40334
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1207/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1207//rebuild

review: Approve (continuous-integration)
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/tests/cloud_tests/platforms/lxd/instance.py b/tests/cloud_tests/platforms/lxd/instance.py
2index 83c97ab..2b804a6 100644
3--- a/tests/cloud_tests/platforms/lxd/instance.py
4+++ b/tests/cloud_tests/platforms/lxd/instance.py
5@@ -4,6 +4,7 @@
6
7 import os
8 import shutil
9+import time
10 from tempfile import mkdtemp
11
12 from cloudinit.util import load_yaml, subp, ProcessExecutionError, which
13@@ -224,7 +225,18 @@ class LXDInstance(Instance):
14 LOG.debug("%s: deleting container.", self)
15 self.unfreeze()
16 self.shutdown()
17- self.pylxd_container.delete(wait=True)
18+ retries = [1] * 5
19+ for attempt, wait in enumerate(retries):
20+ try:
21+ self.pylxd_container.delete(wait=True)
22+ break
23+ except Exception:
24+ if attempt + 1 >= len(retries):
25+ raise
26+ LOG.debug('Failed to delete container %s (%s/%s) retrying...',
27+ self, attempt + 1, len(retries))
28+ time.sleep(wait)
29+
30 self._pylxd_container = None
31
32 if self.platform.container_exists(self.name):

Subscribers

People subscribed via source and target branches