Merge lp:~ltrager/maas-images/image_index into lp:maas-images

Proposed by Lee Trager on 2018-06-11
Status: Merged
Merged at revision: 410
Proposed branch: lp:~ltrager/maas-images/image_index
Merge into: lp:maas-images
Diff against target: 242 lines (+85/-71)
3 files modified
bin/maas-qcow2targz (+17/-6)
conf/centos.yaml (+1/-3)
meph2/commands/mimport.py (+67/-62)
To merge this branch: bzr merge lp:~ltrager/maas-images/image_index
Reviewer Review Type Date Requested Status
Blake Rouse (community) Approve on 2018-09-18
Scott Moser (community) 2018-06-11 Needs Information on 2018-07-09
Review via email: mp+347772@code.launchpad.net

Commit message

Scan for CentOS images using the image-index provided by upstream

When CentOS support was initially added, we decided that CentOS should
follow the same path as Ubuntu. Upstream generates a generic cloud-image
and then lp:maas-images modifies the image to work for MAAS. When this was
first implemented upstream only supplied a sha256sum.txt file[1] for us to
scan for new images. Now, upstream is providing an
image-index [2] file which uses config file format.

The image-index[2] file is missing some older images the sha256sum.txt[1]
has but both contain the latest image file.

[1] https://cloud.centos.org/centos/7/images/sha256sum.txt
[2] https://cloud.centos.org/centos/7/images/image-index

Description of the change

When I initially added CentOS support to lp:maas-images we decided that CentOS should follow the same path as Ubuntu. Upstream generates a generic cloud-image and then lp:maas-images modifies the image to work for MAAS. When this was first implemented upstream only supplied a sha256sum.txt file[1] for us to scan for new images. I noticed that upstream is now providing an image-index [2] file which uses config file format.

The image-index[2] file is missing some older images the sha256sum.txt[1] has but both contain the latest image file.

[1] https://cloud.centos.org/centos/7/images/sha256sum.txt
[2] https://cloud.centos.org/centos/7/images/image-index

To post a comment you must log in.
lp:~ltrager/maas-images/image_index updated on 2018-06-12
401. By Lee Trager on 2018-06-11

Fix comment

402. By Lee Trager on 2018-06-12

Fix exception

Scott Moser (smoser) wrote :

i think there are some logic problems.

review: Needs Information
lp:~ltrager/maas-images/image_index updated on 2018-09-18
403. By Lee Trager on 2018-09-18

Merge trunk

404. By Lee Trager on 2018-09-18

Conform to image-index format

405. By Lee Trager on 2018-09-18

Remove old code

406. By Lee Trager on 2018-09-18

Support upstream release versions

Lee Trager (ltrager) wrote :

Thanks for the review. I've removed the old code as suggested and added support for upstream release versions. I'm ignoring old revisions formats(e.g 20141129) since we aren't building them anyway. While you are correct CentOS 6 and 7 may use the same revision they come from different files[1, 2] which are processed separately.

[1] https://cloud.centos.org/centos/7/images/image-index
[2] https://cloud.centos.org/centos/6/images/image-index

lp:~ltrager/maas-images/image_index updated on 2018-09-18
407. By Lee Trager on 2018-09-18

Remove old check

Blake Rouse (blake-rouse) wrote :

Looks good.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/maas-qcow2targz'
2--- bin/maas-qcow2targz 2018-06-11 18:22:59 +0000
3+++ bin/maas-qcow2targz 2018-09-18 13:41:48 +0000
4@@ -178,12 +178,23 @@
5 error "using cached '${url##*/}' in ${cache_d}"
6 fi
7
8- error "Checking sha256sum for $img_path"
9- sha256_out=$(sha256sum "$img_path") ||
10- fail "failed: sha256sum $img_path"
11- found=${sha256_out% *}
12- if [ "$found" != "$sha256" ]; then
13- fail "Error: unexpected sha256 of $img_path $found != $sha256"
14+ # We were given a SHA512 instead of a SHA256
15+ if [ ${#sha256} -eq 128 ]; then
16+ error "Checking sha512sum for $img_path"
17+ sha512_out=$(sha512sum "$img_path") ||
18+ fail "failed: sha512sum $img_path"
19+ sha512_out=${sha512_out% *}
20+ if [ "$sha512_out" != "$sha256" ]; then
21+ fail "Error: unexpected sha512 of $img_path $sha512_out != $sha256"
22+ fi
23+ else
24+ error "Checking sha256sum for $img_path"
25+ sha256_out=$(sha256sum "$img_path") ||
26+ fail "failed: sha256sum $img_path"
27+ sha256_out=${sha256_out% *}
28+ if [ "$sha256_out" != "$sha256" ]; then
29+ fail "Error: unexpected sha256 of $img_path $sha256_out != $sha256"
30+ fi
31 fi
32
33 fout=$(LANG=C file "$img_path") ||
34
35=== modified file 'conf/centos.yaml'
36--- conf/centos.yaml 2018-09-13 07:55:38 +0000
37+++ conf/centos.yaml 2018-09-18 13:41:48 +0000
38@@ -1,6 +1,4 @@
39-# The closest thing CentOS provides to a machine readable list of images in a
40-# static location is the sha256sum.txt file.
41-sha256_meta_data_path: http://cloud.centos.org/centos/{version}/images/sha256sum.txt
42+image_index: http://cloud.centos.org/centos/{version}/images/image-index
43
44 product_id: "com.ubuntu.maas.daily:centos-bases:{version}:{arch}"
45 content_id: "com.ubuntu.maas:daily:centos-bases-download"
46
47=== modified file 'meph2/commands/mimport.py'
48--- meph2/commands/mimport.py 2018-08-07 21:31:31 +0000
49+++ meph2/commands/mimport.py 2018-09-18 13:41:48 +0000
50@@ -1,10 +1,11 @@
51 #!/usr/bin/python3
52
53+from collections import OrderedDict
54+from configparser import ConfigParser
55+from datetime import datetime
56 import argparse
57-from datetime import datetime
58 import hashlib
59 import os
60-import re
61 import subprocess
62 import sys
63 import yaml
64@@ -19,7 +20,7 @@
65 from meph2.url_helper import geturl_text
66
67
68-def import_sha256(args, product_tree, cfgdata):
69+def import_remote_config(args, product_tree, cfgdata):
70 for (release, release_info) in cfgdata['versions'].items():
71 if 'arch' in release_info:
72 arch = release_info['arch']
73@@ -35,8 +36,20 @@
74 path_version = release_info['version']
75 product_id = cfgdata['product_id'].format(
76 version=release_info['version'], arch=arch)
77- url = cfgdata['sha256_meta_data_path'].format(version=path_version)
78- images = get_sha256_meta_images(url, args.max)
79+ if 'image_index' in cfgdata:
80+ url = cfgdata['image_index'].format(version=path_version)
81+ images_unordered = get_image_index_images(url)
82+ else:
83+ raise ValueError("Undefined remote path")
84+
85+ images = OrderedDict()
86+ if args.max == 0:
87+ max_items = len(images_unordered)
88+ else:
89+ max_items = args.max
90+ for key in sorted(images_unordered.keys(), reverse=True)[:max_items]:
91+ images[key] = images_unordered[key]
92+
93 base_url = os.path.dirname(url)
94
95 if product_tree['products'].get(product_id) is None:
96@@ -52,18 +65,20 @@
97 'versions': {},
98 }
99
100- for (image, image_info) in images.items():
101+ for (revision, image_info) in images.items():
102+ version = '20%s01_%02d' % (revision, image_info['release'])
103 if (
104 product_id in product_tree['products'] and
105- image in product_tree['products'][product_id]['versions']):
106+ version in product_tree['products'][product_id][
107+ 'versions']):
108 print(
109 "Product %s at version %s exists, skipping" % (
110- product_id, image))
111+ product_id, version))
112 continue
113 print(
114 "Downloading and creating %s version %s" % (
115- (product_id, image)))
116- image_path = '/'.join([release, arch, image, 'root-tgz'])
117+ (product_id, version)))
118+ image_path = '/'.join([release, arch, version, 'root-tgz'])
119 real_image_path = os.path.join(
120 os.path.realpath(args.target), image_path)
121 if release_info.get('packages') is not None:
122@@ -71,10 +86,10 @@
123 else:
124 packages = None
125 sha256 = import_qcow2(
126- '/'.join([base_url, image_info['img_name']]),
127- image_info['sha256'], real_image_path,
128+ '/'.join([base_url, image_info['file']]),
129+ image_info['checksum'], real_image_path,
130 release_info.get('curtin_files'), packages)
131- product_tree['products'][product_id]['versions'][image] = {
132+ product_tree['products'][product_id]['versions'][version] = {
133 'items': {
134 'root-image.gz': {
135 'ftype': 'root-tgz',
136@@ -176,54 +191,44 @@
137 }
138
139
140-def get_sha256_meta_images(url, max_items=0):
141- """ Given a URL to a SHA256SUM file return a dictionary of filenames and
142- SHA256 checksums keyed off the file version found as a date string in
143- the filename. This is used in cases where simplestream data isn't
144- avalible.
145+def get_image_index_images(url):
146+ """ Given a URL to an image-index config file return a dictionary of
147+ filenames and SHA256 checksums keyed off the revision.
148 """
149 ret = dict()
150 content = geturl_text(url)
151- # http://cloud.centos.org/centos/ contains images using two version
152- # strings. The first is only used on older images and uses the format
153- # YYYYMMDD_XX. The second is used on images generated monthly using the
154- # format YYMM. We know the second format is referencing the year and month
155- # by looking at the timestamp of each image.
156- prog = re.compile('([\d]{8}(_[\d]+))|(\d{4})')
157-
158- for i in content.split('\n'):
159- try:
160- sha256, img_name = i.split()
161- except ValueError:
162- continue
163- if (not img_name.endswith('qcow2.xz') and
164- not img_name.endswith('qcow2')):
165- continue
166- m = prog.search(img_name)
167- if m is None:
168- continue
169- img_version = m.group(0)
170-
171- # Turn the short version string into a long version string so that MAAS
172- # uses the latest version, not the longest
173- if len(img_version) == 4:
174- img_version = "20%s01_01" % img_version
175-
176- # Prefer compressed image over uncompressed
177- if (img_version in ret and
178- ret[img_version]['img_name'].endswith('qcow2.xz')):
179- continue
180- ret[img_version] = {
181- 'img_name': img_name,
182- 'sha256': sha256,
183- }
184- if max_items == 0:
185- return ret
186- else:
187- return {
188- key: ret[key]
189- for key in sorted(ret.keys(), reverse=True)[:max_items]
190- }
191+ config = ConfigParser()
192+ config.read_string(content)
193+ for section in config.values():
194+ # ConfigParser defines a 'DEFAULT' section with nothing in it...
195+ if section.name == 'DEFAULT':
196+ continue
197+ skip = False
198+ for required_key in ['name', 'file', 'revision', 'checksum']:
199+ if required_key not in section:
200+ sys.stderr.write(
201+ "'%s' is undefined in section %s, skipping!\n" % (
202+ required_key, section.name))
203+ skip = True
204+ if skip:
205+ continue
206+
207+ revision = section.get('revision')
208+ if '_' in revision:
209+ revision, release = revision.split('_')
210+ elif '-' in revision:
211+ revision, release = revision.split('-')
212+ else:
213+ release = 1
214+
215+ # Ignore old unsupported revision format(e.g 20150628_01)
216+ if len(revision) != 4:
217+ continue
218+
219+ ret[revision] = dict(section)
220+ ret[revision]['release'] = release
221+
222+ return ret
223
224
225 def import_qcow2(url, expected_sha256, out, curtin_files=None, packages=None):
226@@ -280,12 +285,12 @@
227 product_tree['updated'] = util.timestamp()
228 product_tree['datatype'] = 'image-downloads'
229
230- if cfgdata.get('sha256_meta_data_path', None) is not None:
231- import_sha256(args, product_tree, cfgdata)
232- elif cfgdata.get('bootloaders', None) is not None:
233+ if cfgdata.get('image_index') is not None:
234+ import_remote_config(args, product_tree, cfgdata)
235+ elif cfgdata.get('bootloaders') is not None:
236 import_bootloaders(args, product_tree, cfgdata)
237 else:
238- sys.stderr.write('Unsupported import yaml!')
239+ sys.stderr.write('Unsupported import yaml!\n')
240 sys.exit(1)
241
242 md_d = os.path.join(args.target, 'streams', 'v1')

Subscribers

People subscribed via source and target branches