curtin

Merge lp:~raharper/curtin/sync-images-convert into lp:~curtin-dev/curtin/trunk

  1. sync-images-convert
  2. Merge into trunk
Proposed by Ryan Harper
Status: Merged
Merged at revision: 337
Proposed branch: lp:~raharper/curtin/sync-images-convert
Merge into: lp:~curtin-dev/curtin/trunk
Diff against target: 98 lines (+38/-7)
2 files modified
tests/vmtests/__init__.py (+29/-5)
tools/vmtest-sync-images (+9/-2)
To merge this branch: bzr merge lp:~raharper/curtin/sync-images-convert
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve
curtin developers Pending
Review via email: mp+282021@code.launchpad.net

Description of the change

sync-images: auto-convert and prune images

make sync-images will now ensure we invoke convert_image() as needed on each one and we now have a default of 1 image to keep around. This number is controlled by an environment variable, IMAGES_TO_KEEP.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:340
https://server-team-jenkins.canonical.com/job/curtin-ci/107/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/104/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/107/rebuild

review: Approve (continuous-integration)
lp:~raharper/curtin/sync-images-convert updated
341. By Ryan Harper

sstream-mirror use IMAGES_TO_KEEP for --max

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:341
https://server-team-jenkins.canonical.com/job/curtin-ci/109/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/106/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/109/rebuild

review: Approve (continuous-integration)
lp:~raharper/curtin/sync-images-convert updated
342. By Ryan Harper

convert IMAGES_TO_KEEP to int

343. By Ryan Harper

Only use --max for mirror, query needs max=1

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:343
https://server-team-jenkins.canonical.com/job/curtin-ci/110/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/107/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/110/rebuild

review: Approve (continuous-integration)
lp:~raharper/curtin/sync-images-convert updated
344. By Ryan Harper

max can't be in the default or mirror gets it twice

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:344
https://server-team-jenkins.canonical.com/job/curtin-ci/111/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/108/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/111/rebuild

review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py 2015-12-18 20:25:09 +0000
+++ tests/vmtests/__init__.py 2016-01-08 17:11:34 +0000
@@ -25,9 +25,15 @@
25 "http://maas.ubuntu.com/images/ephemeral-v2/daily/streams/v1/index.sjson")25 "http://maas.ubuntu.com/images/ephemeral-v2/daily/streams/v1/index.sjson")
2626
27IMAGE_DIR = os.environ.get("IMAGE_DIR", "/srv/images")27IMAGE_DIR = os.environ.get("IMAGE_DIR", "/srv/images")
28try:
29 IMAGES_TO_KEEP = int(os.environ.get("IMAGES_TO_KEEP", 1))
30except ValueError:
31 raise ValueError("IMAGES_TO_KEEP in environment was not an integer")
32
28DEFAULT_SSTREAM_OPTS = [33DEFAULT_SSTREAM_OPTS = [
29 '--max=1', '--keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg']34 '--keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg']
30DEFAULT_FILTERS = ['arch=amd64', 'item_name=root-image.gz']35DEFAULT_ARCH = 'amd64'
36DEFAULT_FILTERS = ['arch=%s' % DEFAULT_ARCH, 'item_name=root-image.gz']
3137
32DEVNULL = open(os.devnull, 'w')38DEVNULL = open(os.devnull, 'w')
33KEEP_DATA = {"pass": "none", "fail": "all"}39KEEP_DATA = {"pass": "none", "fail": "all"}
@@ -47,6 +53,11 @@
47 pass53 pass
4854
4955
56def remove_dir(dirpath):
57 if os.path.exists(dirpath):
58 shutil.rmtree(dirpath)
59
60
50def _topdir():61def _topdir():
51 global _TOPDIR62 global _TOPDIR
52 if _TOPDIR:63 if _TOPDIR:
@@ -172,12 +183,25 @@
172 # then the real sstream-mirror call below will also fail.183 # then the real sstream-mirror call below will also fail.
173 pass184 pass
174185
175 cmd = ['sstream-mirror'] + DEFAULT_SSTREAM_OPTS + progress + [186 cmd = (['sstream-mirror'] + DEFAULT_SSTREAM_OPTS + progress + [
176 self.source_url, self.base_dir] + filters187 '--max=%s' % IMAGES_TO_KEEP, self.source_url, self.base_dir] +
188 filters)
177 logger.info('Syncing images {}'.format(cmd))189 logger.info('Syncing images {}'.format(cmd))
178 out = subprocess.check_output(cmd)190 out = subprocess.check_output(cmd)
179 logger.debug(out)191 logger.debug(out)
180192
193 def prune_images(self, release, arch, filters=None):
194 image_dir = os.path.join(self.base_dir, release, arch)
195 release_dirs = sorted(os.listdir(image_dir))
196 logger.info('Pruning release={} keep={}'.format(release,
197 IMAGES_TO_KEEP))
198 if len(release_dirs) > IMAGES_TO_KEEP:
199 to_remove = release_dirs[0:-IMAGES_TO_KEEP]
200 logger.info('Removing {} images'.format(len(to_remove)))
201 for d in to_remove:
202 fullpath = os.path.join(image_dir, d)
203 remove_dir(fullpath)
204
181 def get_image(self, release, arch, filters=None):205 def get_image(self, release, arch, filters=None):
182 """Return local path for root image, kernel and initrd, tarball."""206 """Return local path for root image, kernel and initrd, tarball."""
183 if filters is None:207 if filters is None:
@@ -187,7 +211,7 @@
187 logger.info(211 logger.info(
188 'Query simplestreams for root image: %s', filters)212 'Query simplestreams for root image: %s', filters)
189 cmd = ['sstream-query'] + DEFAULT_SSTREAM_OPTS + [213 cmd = ['sstream-query'] + DEFAULT_SSTREAM_OPTS + [
190 self.url, 'item_name=root-image.gz'] + filters214 '--max=1', self.url, 'item_name=root-image.gz'] + filters
191 logger.debug(" ".join(cmd))215 logger.debug(" ".join(cmd))
192 out = subprocess.check_output(cmd)216 out = subprocess.check_output(cmd)
193 logger.debug(out)217 logger.debug(out)
194218
=== modified file 'tools/vmtest-sync-images'
--- tools/vmtest-sync-images 2015-11-05 19:44:39 +0000
+++ tools/vmtest-sync-images 2016-01-08 17:11:34 +0000
@@ -8,14 +8,21 @@
8sys.path.append(os.path.join(os.path.dirname(__file__), '..'))8sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
99
10from tests.vmtests import (10from tests.vmtests import (
11 ImageStore, IMAGE_DIR, IMAGE_SRC_URL, DEFAULT_FILTERS)11 ImageStore, IMAGE_DIR, IMAGE_SRC_URL, DEFAULT_FILTERS, DEFAULT_ARCH)
12from tests.vmtests.helpers import find_releases12from tests.vmtests.helpers import find_releases
1313
1414
15if __name__ == '__main__':15if __name__ == '__main__':
16 # Instantiate the ImageStore object.16 # Instantiate the ImageStore object.
17 store = ImageStore(IMAGE_SRC_URL, IMAGE_DIR)17 store = ImageStore(IMAGE_SRC_URL, IMAGE_DIR)
18 release_filter = 'release~{}'.format('|'.join(find_releases()))18 releases = find_releases()
19 release_filter = 'release~{}'.format('|'.join(releases))
19 DEFAULT_FILTERS.append(release_filter)20 DEFAULT_FILTERS.append(release_filter)
20 # Sync images.21 # Sync images.
21 store.sync_images(filters=DEFAULT_FILTERS)22 store.sync_images(filters=DEFAULT_FILTERS)
23 # Drop old images and convert new images.
24 for release in releases:
25 store.prune_images(release, DEFAULT_ARCH)
26 store.get_image(release, DEFAULT_ARCH)
27
28

Subscribers

People subscribed via source and target branches