Merge lp:~raharper/curtin/sync-images-convert into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

sync-images: auto-convert and prune images

make sync-images will now ensure we invoke convert_image() as needed on each one and we now have a default of 1 image to keep around. This number is controlled by an environment variable, IMAGES_TO_KEEP.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:340
https://server-team-jenkins.canonical.com/job/curtin-ci/107/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/104/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/107/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/sync-images-convert]

341. By Ryan Harper on 2016-01-08

sstream-mirror use IMAGES_TO_KEEP for --max

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:341
https://server-team-jenkins.canonical.com/job/curtin-ci/109/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/106/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/109/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/sync-images-convert]

342. By Ryan Harper on 2016-01-08

convert IMAGES_TO_KEEP to int

343. By Ryan Harper on 2016-01-08

Only use --max for mirror, query needs max=1

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:343
https://server-team-jenkins.canonical.com/job/curtin-ci/110/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/107/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/110/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/sync-images-convert]

344. By Ryan Harper on 2016-01-08

max can't be in the default or mirror gets it twice

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:344
https://server-team-jenkins.canonical.com/job/curtin-ci/111/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/108/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/111/rebuild

review: Approve (continuous-integration)

=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py	2015-12-18 20:25:09 +0000
+++ tests/vmtests/__init__.py	2016-01-08 17:11:34 +0000
@@ -25,9 +25,15 @@
     "http://maas.ubuntu.com/images/ephemeral-v2/daily/streams/v1/index.sjson")
 
 IMAGE_DIR = os.environ.get("IMAGE_DIR", "/srv/images")
+try:
+    IMAGES_TO_KEEP = int(os.environ.get("IMAGES_TO_KEEP", 1))
+except ValueError:
+    raise ValueError("IMAGES_TO_KEEP in environment was not an integer")
+
 DEFAULT_SSTREAM_OPTS = [
-    '--max=1', '--keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg']
-DEFAULT_FILTERS = ['arch=amd64', 'item_name=root-image.gz']
+    '--keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg']
+DEFAULT_ARCH = 'amd64'
+DEFAULT_FILTERS = ['arch=%s' % DEFAULT_ARCH, 'item_name=root-image.gz']
 
 DEVNULL = open(os.devnull, 'w')
 KEEP_DATA = {"pass": "none", "fail": "all"}
@@ -47,6 +53,11 @@
                 pass
 
 
+def remove_dir(dirpath):
+    if os.path.exists(dirpath):
+        shutil.rmtree(dirpath)
+
+
 def _topdir():
     global _TOPDIR
     if _TOPDIR:
@@ -172,12 +183,25 @@
             # then the real sstream-mirror call below will also fail.
             pass
 
-        cmd = ['sstream-mirror'] + DEFAULT_SSTREAM_OPTS + progress + [
-            self.source_url, self.base_dir] + filters
+        cmd = (['sstream-mirror'] + DEFAULT_SSTREAM_OPTS + progress + [
+            '--max=%s' % IMAGES_TO_KEEP, self.source_url, self.base_dir] +
+            filters)
         logger.info('Syncing images {}'.format(cmd))
         out = subprocess.check_output(cmd)
         logger.debug(out)
 
+    def prune_images(self, release, arch, filters=None):
+        image_dir = os.path.join(self.base_dir, release, arch)
+        release_dirs = sorted(os.listdir(image_dir))
+        logger.info('Pruning release={} keep={}'.format(release,
+                                                        IMAGES_TO_KEEP))
+        if len(release_dirs) > IMAGES_TO_KEEP:
+            to_remove = release_dirs[0:-IMAGES_TO_KEEP]
+            logger.info('Removing {} images'.format(len(to_remove)))
+            for d in to_remove:
+                fullpath = os.path.join(image_dir, d)
+                remove_dir(fullpath)
+
     def get_image(self, release, arch, filters=None):
         """Return local path for root image, kernel and initrd, tarball."""
         if filters is None:
@@ -187,7 +211,7 @@
         logger.info(
             'Query simplestreams for root image: %s', filters)
         cmd = ['sstream-query'] + DEFAULT_SSTREAM_OPTS + [
-            self.url, 'item_name=root-image.gz'] + filters
+            '--max=1', self.url, 'item_name=root-image.gz'] + filters
         logger.debug(" ".join(cmd))
         out = subprocess.check_output(cmd)
         logger.debug(out)

=== modified file 'tools/vmtest-sync-images'
--- tools/vmtest-sync-images	2015-11-05 19:44:39 +0000
+++ tools/vmtest-sync-images	2016-01-08 17:11:34 +0000
@@ -8,14 +8,21 @@
 sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
 
 from tests.vmtests import (
-    ImageStore, IMAGE_DIR, IMAGE_SRC_URL, DEFAULT_FILTERS)
+    ImageStore, IMAGE_DIR, IMAGE_SRC_URL, DEFAULT_FILTERS, DEFAULT_ARCH)
 from tests.vmtests.helpers import find_releases
 
 
 if __name__ == '__main__':
     # Instantiate the ImageStore object.
     store = ImageStore(IMAGE_SRC_URL, IMAGE_DIR)
-    release_filter = 'release~{}'.format('|'.join(find_releases()))
+    releases = find_releases()
+    release_filter = 'release~{}'.format('|'.join(releases))
     DEFAULT_FILTERS.append(release_filter)
     # Sync images.
     store.sync_images(filters=DEFAULT_FILTERS)
+    # Drop old images and convert new images.
+    for release in releases:
+        store.prune_images(release, DEFAULT_ARCH)
+        store.get_image(release, DEFAULT_ARCH)
+
+