Merge ~raharper/cloud-init:fix/cloud-tests-lxd-retry-delete into cloud-init:master

[Ryan Harper]

cloud_test/lxd: Retry container delete a few times

LXD integration tests fail sometimes due to failure to delete the
container, usually related to ZFS backend. This is a transient
issue unrelated to the test itself. Teach LXD platform to retry
this a few times before returning an error.

[Paride Legovini (paride)]

LGTM

review: Approve

[Ryan Harper (raharper)]

I need to add a sleep after the failure

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:c6bdca59c2b7afdf19b9bc7a4df8367584016627
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1206/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1206//rebuild

review: Needs Fixing (continuous-integration)

aaf0c0e... by Ryan Harper on 2019-10-11

Add the sleep before next retry.

[Paride Legovini (paride)]

Right. In the usual failure mode retrying should work even without sleeping, but it's way better to add a sleep(1) there.

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:aaf0c0e11ed07b65d15f8759d907483561c40334
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1207/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1207//rebuild

review: Approve (continuous-integration)

[Dan Watkins (oddbloke)]

review: Approve

diff --git a/tests/cloud_tests/platforms/lxd/instance.py b/tests/cloud_tests/platforms/lxd/instance.py
index 83c97ab..2b804a6 100644
--- a/tests/cloud_tests/platforms/lxd/instance.py
+++ b/tests/cloud_tests/platforms/lxd/instance.py
@@ -4,6 +4,7 @@
 
 import os
 import shutil
+import time
 from tempfile import mkdtemp
 
 from cloudinit.util import load_yaml, subp, ProcessExecutionError, which
@@ -224,7 +225,18 @@ class LXDInstance(Instance):
         LOG.debug("%s: deleting container.", self)
         self.unfreeze()
         self.shutdown()
-        self.pylxd_container.delete(wait=True)
+        retries = [1] * 5
+        for attempt, wait in enumerate(retries):
+            try:
+                self.pylxd_container.delete(wait=True)
+                break
+            except Exception:
+                if attempt + 1 >= len(retries):
+                    raise
+                LOG.debug('Failed to delete container %s (%s/%s) retrying...',
+                          self, attempt + 1, len(retries))
+                time.sleep(wait)
+
         self._pylxd_container = None
 
         if self.platform.container_exists(self.name):