Merge lp:~ltrager/maas-images/image_index into lp:maas-images

[Lee Trager]

Scan for CentOS images using the image-index provided by upstream

When CentOS support was initially added, we decided that CentOS should
follow the same path as Ubuntu. Upstream generates a generic cloud-image
and then lp:maas-images modifies the image to work for MAAS. When this was
first implemented upstream only supplied a sha256sum.txt file[1] for us to
scan for new images. Now, upstream is providing an
image-index [2] file which uses config file format.

The image-index[2] file is missing some older images the sha256sum.txt[1]
has but both contain the latest image file.

[1] https://cloud.centos.org/centos/7/images/sha256sum.txt
[2] https://cloud.centos.org/centos/7/images/image-index

When I initially added CentOS support to lp:maas-images we decided that CentOS should follow the same path as Ubuntu. Upstream generates a generic cloud-image and then lp:maas-images modifies the image to work for MAAS. When this was first implemented upstream only supplied a sha256sum.txt file[1] for us to scan for new images. I noticed that upstream is now providing an image-index [2] file which uses config file format.

The image-index[2] file is missing some older images the sha256sum.txt[1] has but both contain the latest image file.

[1] https://cloud.centos.org/centos/7/images/sha256sum.txt
[2] https://cloud.centos.org/centos/7/images/image-index

[lp:~ltrager/maas-images/image_index]

401. By Lee Trager on 2018-06-11

Fix comment

402. By Lee Trager on 2018-06-12

Fix exception

[Scott Moser (smoser)]

i think there are some logic problems.

review: Needs Information

[lp:~ltrager/maas-images/image_index]

403. By Lee Trager on 2018-09-18

Merge trunk

404. By Lee Trager on 2018-09-18

Conform to image-index format

405. By Lee Trager on 2018-09-18

Remove old code

406. By Lee Trager on 2018-09-18

Support upstream release versions

[Lee Trager (ltrager)]

Thanks for the review. I've removed the old code as suggested and added support for upstream release versions. I'm ignoring old revisions formats(e.g 20141129) since we aren't building them anyway. While you are correct CentOS 6 and 7 may use the same revision they come from different files[1, 2] which are processed separately.

[1] https://cloud.centos.org/centos/7/images/image-index
[2] https://cloud.centos.org/centos/6/images/image-index

[lp:~ltrager/maas-images/image_index]

407. By Lee Trager on 2018-09-18

Remove old check

[Blake Rouse (blake-rouse)]

Looks good.

review: Approve

=== modified file 'bin/maas-qcow2targz'
--- bin/maas-qcow2targz	2018-06-11 18:22:59 +0000
+++ bin/maas-qcow2targz	2018-09-18 13:41:48 +0000
@@ -178,12 +178,23 @@
             error "using cached '${url##*/}' in ${cache_d}"
         fi
 
-        error "Checking sha256sum for $img_path"
-        sha256_out=$(sha256sum "$img_path") ||
-            fail "failed: sha256sum $img_path"
-        found=${sha256_out%  *}
-        if [ "$found" != "$sha256" ]; then
-            fail "Error: unexpected sha256 of $img_path $found != $sha256"
+        # We were given a SHA512 instead of a SHA256
+        if [ ${#sha256} -eq 128 ]; then
+            error "Checking sha512sum for $img_path"
+            sha512_out=$(sha512sum "$img_path") ||
+                fail "failed: sha512sum $img_path"
+            sha512_out=${sha512_out%  *}
+            if [ "$sha512_out" != "$sha256" ]; then
+                fail "Error: unexpected sha512 of $img_path $sha512_out != $sha256"
+            fi
+        else
+            error "Checking sha256sum for $img_path"
+            sha256_out=$(sha256sum "$img_path") ||
+                fail "failed: sha256sum $img_path"
+            sha256_out=${sha256_out%  *}
+            if [ "$sha256_out" != "$sha256" ]; then
+                fail "Error: unexpected sha256 of $img_path $sha256_out != $sha256"
+            fi
         fi
 
         fout=$(LANG=C file "$img_path") ||

=== modified file 'conf/centos.yaml'
--- conf/centos.yaml	2018-09-13 07:55:38 +0000
+++ conf/centos.yaml	2018-09-18 13:41:48 +0000
@@ -1,6 +1,4 @@
-# The closest thing CentOS provides to a machine readable list of images in a
-# static location is the sha256sum.txt file.
-sha256_meta_data_path: http://cloud.centos.org/centos/{version}/images/sha256sum.txt
+image_index: http://cloud.centos.org/centos/{version}/images/image-index
 
 product_id: "com.ubuntu.maas.daily:centos-bases:{version}:{arch}"
 content_id: "com.ubuntu.maas:daily:centos-bases-download"

=== modified file 'meph2/commands/mimport.py'
--- meph2/commands/mimport.py	2018-08-07 21:31:31 +0000
+++ meph2/commands/mimport.py	2018-09-18 13:41:48 +0000
@@ -1,10 +1,11 @@
 #!/usr/bin/python3
 
+from collections import OrderedDict
+from configparser import ConfigParser
+from datetime import datetime
 import argparse
-from datetime import datetime
 import hashlib
 import os
-import re
 import subprocess
 import sys
 import yaml
@@ -19,7 +20,7 @@
 from meph2.url_helper import geturl_text
 
 
-def import_sha256(args, product_tree, cfgdata):
+def import_remote_config(args, product_tree, cfgdata):
     for (release, release_info) in cfgdata['versions'].items():
         if 'arch' in release_info:
             arch = release_info['arch']
@@ -35,8 +36,20 @@
             path_version = release_info['version']
         product_id = cfgdata['product_id'].format(
             version=release_info['version'], arch=arch)
-        url = cfgdata['sha256_meta_data_path'].format(version=path_version)
-        images = get_sha256_meta_images(url, args.max)
+        if 'image_index' in cfgdata:
+            url = cfgdata['image_index'].format(version=path_version)
+            images_unordered = get_image_index_images(url)
+        else:
+            raise ValueError("Undefined remote path")
+
+        images = OrderedDict()
+        if args.max == 0:
+            max_items = len(images_unordered)
+        else:
+            max_items = args.max
+        for key in sorted(images_unordered.keys(), reverse=True)[:max_items]:
+            images[key] = images_unordered[key]
+
         base_url = os.path.dirname(url)
 
         if product_tree['products'].get(product_id) is None:
@@ -52,18 +65,20 @@
                 'versions': {},
             }
 
-        for (image, image_info) in images.items():
+        for (revision, image_info) in images.items():
+            version = '20%s01_%02d' % (revision, image_info['release'])
             if (
                     product_id in product_tree['products'] and
-                    image in product_tree['products'][product_id]['versions']):
+                    version in product_tree['products'][product_id][
+                        'versions']):
                 print(
                     "Product %s at version %s exists, skipping" % (
-                        product_id, image))
+                        product_id, version))
                 continue
             print(
                 "Downloading and creating %s version %s" % (
-                    (product_id, image)))
-            image_path = '/'.join([release, arch, image, 'root-tgz'])
+                    (product_id, version)))
+            image_path = '/'.join([release, arch, version, 'root-tgz'])
             real_image_path = os.path.join(
                 os.path.realpath(args.target), image_path)
             if release_info.get('packages') is not None:
@@ -71,10 +86,10 @@
             else:
                 packages = None
             sha256 = import_qcow2(
-                '/'.join([base_url, image_info['img_name']]),
-                image_info['sha256'], real_image_path,
+                '/'.join([base_url, image_info['file']]),
+                image_info['checksum'], real_image_path,
                 release_info.get('curtin_files'), packages)
-            product_tree['products'][product_id]['versions'][image] = {
+            product_tree['products'][product_id]['versions'][version] = {
                 'items': {
                     'root-image.gz': {
                         'ftype': 'root-tgz',
@@ -176,54 +191,44 @@
         }
 
 
-def get_sha256_meta_images(url, max_items=0):
-    """ Given a URL to a SHA256SUM file return a dictionary of filenames and
-        SHA256 checksums keyed off the file version found as a date string in
-        the filename. This is used in cases where simplestream data isn't
-        avalible.
+def get_image_index_images(url):
+    """ Given a URL to an image-index config file return a dictionary of
+        filenames and SHA256 checksums keyed off the revision.
     """
     ret = dict()
     content = geturl_text(url)
-    # http://cloud.centos.org/centos/ contains images using two version
-    # strings. The first is only used on older images and uses the format
-    # YYYYMMDD_XX. The second is used on images generated monthly using the
-    # format YYMM. We know the second format is referencing the year and month
-    # by looking at the timestamp of each image.
-    prog = re.compile('([\d]{8}(_[\d]+))|(\d{4})')
-
-    for i in content.split('\n'):
-        try:
-            sha256, img_name = i.split()
-        except ValueError:
-            continue
-        if (not img_name.endswith('qcow2.xz') and
-                not img_name.endswith('qcow2')):
-            continue
-        m = prog.search(img_name)
-        if m is None:
-            continue
-        img_version = m.group(0)
-
-        # Turn the short version string into a long version string so that MAAS
-        # uses the latest version, not the longest
-        if len(img_version) == 4:
-            img_version = "20%s01_01" % img_version
-
-        # Prefer compressed image over uncompressed
-        if (img_version in ret and
-                ret[img_version]['img_name'].endswith('qcow2.xz')):
-            continue
-        ret[img_version] = {
-            'img_name': img_name,
-            'sha256': sha256,
-            }
-    if max_items == 0:
-        return ret
-    else:
-        return {
-            key: ret[key]
-            for key in sorted(ret.keys(), reverse=True)[:max_items]
-        }
+    config = ConfigParser()
+    config.read_string(content)
+    for section in config.values():
+        # ConfigParser defines a 'DEFAULT' section with nothing in it...
+        if section.name == 'DEFAULT':
+            continue
+        skip = False
+        for required_key in ['name', 'file', 'revision', 'checksum']:
+            if required_key not in section:
+                sys.stderr.write(
+                    "'%s' is undefined in section %s, skipping!\n" % (
+                        required_key, section.name))
+                skip = True
+        if skip:
+            continue
+
+        revision = section.get('revision')
+        if '_' in revision:
+            revision, release = revision.split('_')
+        elif '-' in revision:
+            revision, release = revision.split('-')
+        else:
+            release = 1
+
+        # Ignore old unsupported revision format(e.g 20150628_01)
+        if len(revision) != 4:
+            continue
+
+        ret[revision] = dict(section)
+        ret[revision]['release'] = release
+
+    return ret
 
 
 def import_qcow2(url, expected_sha256, out, curtin_files=None, packages=None):
@@ -280,12 +285,12 @@
     product_tree['updated'] = util.timestamp()
     product_tree['datatype'] = 'image-downloads'
 
-    if cfgdata.get('sha256_meta_data_path', None) is not None:
-        import_sha256(args, product_tree, cfgdata)
-    elif cfgdata.get('bootloaders', None) is not None:
+    if cfgdata.get('image_index') is not None:
+        import_remote_config(args, product_tree, cfgdata)
+    elif cfgdata.get('bootloaders') is not None:
         import_bootloaders(args, product_tree, cfgdata)
     else:
-        sys.stderr.write('Unsupported import yaml!')
+        sys.stderr.write('Unsupported import yaml!\n')
         sys.exit(1)
 
     md_d = os.path.join(args.target, 'streams', 'v1')