Merge lp:~ltrager/maas-images/bootloaders into lp:maas-images

[Lee Trager]

Import and publish bootloaders from a yaml file.

This allows the import subcommand to accept a yaml file containing a list of bootloaders. The latest version of the bootloader is downloaded from the Ubuntu archive. Files can be copied out of the bootloader or they can be processed with grub-mkimage.

This also fixes a bug with the import command where if any version of a product existed no newer version would be imported. This will allow more up-to-date CentOS images to be published.

This allows bootloaders to be published in the MAAS SimpleStream. As this currently breaks the MAAS import process, due to missing fields, boot loaders will only be added to http://images.maas.io/ephemeral-v3/

Example product listing: http://paste.ubuntu.com/20006681/

[lp:~ltrager/maas-images/bootloaders]

309. By Lee Trager on 2016-07-19

Fix PEP8 errors

310. By Lee Trager on 2016-07-19

We don't need a boot_loader entry, we already have the ftype

[Blake Rouse (blake-rouse)]

See my comment about the release for the bootloaders. I have not looked closely at the code yet.

[lp:~ltrager/maas-images/bootloaders]

311. By Lee Trager on 2016-07-21

blake_r corrections

[Lee Trager (ltrager)]

The bootloader import process now iterates through updates-<release>, security-<release>, and <release> to find the latest version of the package. This ensures we will always be publishing the latest version that is in the Ubuntu archives.

I also added the ability for an individual bootloader to override the global release. This allows us to pull bootloaders from different releases. Right now they all pull from Xenial.

[lp:~ltrager/maas-images/bootloaders]

312. By Lee Trager on 2016-07-21

Make each bootloader specify a release

[Lee Trager (ltrager)]

After thinking about this we should just make each bootloader specify a release. We already require each to specify an archive.

=== added file 'conf/bootloaders.yaml'
--- conf/bootloaders.yaml	1970-01-01 00:00:00 +0000
+++ conf/bootloaders.yaml	2016-07-21 22:02:46 +0000
@@ -0,0 +1,45 @@
+product_id: "com.ubuntu.maas.daily:bootloaders-bases:{bootloader}"
+content_id: "com.ubuntu.maas:daily:bootloaders-bases-download"
+
+bootloaders:
+ - bootloader: pxe
+   packages:
+    - pxelinux
+    - syslinux-common
+   arch: amd64
+   archive: http://archive.ubuntu.com/ubuntu
+   release: xenial
+   files:
+    - /usr/lib/PXELINUX/pxelinux.0
+    - /usr/lib/syslinux/modules/bios/*.c32
+ - bootloader: uefi
+   packages:
+    - shim-signed
+    - grub-efi-amd64-signed
+   arch: amd64
+   archive: http://archive.ubuntu.com/ubuntu
+   release: xenial
+   files:
+    - /usr/lib/shim/shim.efi.signed
+    - /usr/lib/grub/x86_64-efi-signed/grubnetx64.efi.signed
+    - /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
+ - bootloader: uefi-arm64
+   packages:
+    - grub-efi-arm64-bin
+   arch: arm64
+   archive: http://ports.ubuntu.com
+   release: xenial
+   files:
+    - /usr/lib/grub/arm64-efi/
+   grub_format: arm64-efi
+   grub_output: grubaa64.efi
+ - bootloader: powerkvm
+   packages:
+    - grub-ieee1275-bin
+   arch: ppc64el
+   archive: http://ports.ubuntu.com
+   release: xenial
+   files:
+    - /usr/lib/grub/powerpc-ieee1275/
+   grub_format: powerpc-ieee1275
+   grub_output: bootppc64.bin

=== added file 'meph2/commands/dpkg.py'
--- meph2/commands/dpkg.py	1970-01-01 00:00:00 +0000
+++ meph2/commands/dpkg.py	2016-07-21 22:02:46 +0000
@@ -0,0 +1,183 @@
+from platform import linux_distribution
+import apt_pkg
+import shutil
+import subprocess
+import hashlib
+import io
+import lzma
+import os
+import re
+import sys
+import tempfile
+import urllib.request
+import glob
+
+
+def get_distro_release():
+    """Returns the release name for the running distro."""
+    disname, version, codename = linux_distribution()
+    return codename
+
+
+def get_file(url):
+    """Downloads the file from the given URL into memory.
+
+    :param url" URL to download
+    :return: File data, or None
+    """
+    # Build a newer opener so that the environment is checked for proxy
+    # URLs. Using urllib2.urlopen() means that we'd only be using the
+    # proxies as defined when urlopen() was called the first time.
+    try:
+        response = urllib.request.build_opener().open(url)
+        return response.read()
+    except urllib.error.URLError as e:
+        sys.stderr.write("Unable to download %s: %s" % (url, str(e.reason)))
+        sys.exit(1)
+    except BaseException as e:
+        sys.stderr.write("Unable to download %s: %s" % (url, str(e)))
+        sys.exit(1)
+
+
+def get_sha256(data):
+    """Returns the SHA256SUM for the provided data."""
+    sha256 = hashlib.sha256()
+    sha256.update(data)
+    return sha256.hexdigest()
+
+
+def gpg_verify_data(signature, data_file):
+    """Verify's data using the signature."""
+    tmp = tempfile.mkdtemp(prefix='maas-images')
+    sig_out = os.path.join(tmp, 'verify.gpg')
+    with open(sig_out, 'wb') as stream:
+        stream.write(signature)
+
+    data_out = os.path.join(tmp, 'verify')
+    with open(data_out, 'wb') as stream:
+        stream.write(data_file)
+
+    subprocess.check_output(
+        ['gpgv', '--keyring', '/etc/apt/trusted.gpg', sig_out, data_out],
+        stderr=subprocess.STDOUT)
+
+    shutil.rmtree(tmp, ignore_errors=True)
+
+
+# Cache packages
+_packages = {}
+
+
+def get_packages(base_url, architecture, pkg_name):
+    """Gets the package list from the archive verified."""
+    global _packages
+    release_url = '%s/%s' % (base_url, 'Release')
+    path = 'main/binary-%s/Packages.xz' % architecture
+    packages_url = '%s/%s' % (base_url, path)
+    if packages_url in _packages:
+        return _packages[packages_url]
+    release_file = get_file(release_url)
+    release_file_gpg = get_file('%s.gpg' % release_url)
+    gpg_verify_data(release_file_gpg, release_file)
+
+    # Download the packages file and verify the SHA256SUM
+    pkg_data = get_file(packages_url)
+    regex_path = re.escape(path)
+    sha256sum = re.search(
+        ("^\s*?([a-fA-F0-9]{64})\s*[0-9]+\s+%s$" % regex_path).encode('utf-8'),
+        release_file,
+        re.MULTILINE).group(1)
+    if get_sha256(pkg_data).encode('utf-8') != sha256sum:
+        sys.stderr.write("Unable to verify %s" % packages_url)
+        sys.exit(1)
+
+    _packages[packages_url] = {}
+    compressed = io.BytesIO(pkg_data)
+    with lzma.LZMAFile(compressed) as uncompressed:
+        pkg_name = None
+        package = {}
+        for line in uncompressed:
+            line = line.decode('utf-8')
+            if line == '\n':
+                _packages[packages_url][pkg_name] = package
+                pkg_name = None
+                package = {}
+                continue
+            key, value = line.split(': ', 1)
+            value = value.strip()
+            if key == 'Package':
+                pkg_name = value
+            else:
+                package[key] = value
+
+    return _packages[packages_url]
+
+
+def get_package(archive, pkg_name, architecture, release=None, dest=None):
+    """Look through the archives for package metadata. If a dest is given
+    download the package.
+
+    :return: A dictionary containing the packages meta info or if a dest is
+             given the path of the downloaded file."""
+    global _packages
+    release = get_distro_release() if release is None else release
+    package = None
+    apt_pkg.init()
+    # Find the latest version of the package
+    for dist in ('%s-updates' % release, '%s-security' % release, release):
+        base_url = '%s/dists/%s' % (archive, dist)
+        packages = get_packages(base_url, architecture, pkg_name)
+        if pkg_name in packages:
+            if package is None or apt_pkg.version_compare(
+                    packages[pkg_name]['Version'], package['Version']) > 0:
+                package = packages[pkg_name]
+    # Download it if it was found and a dest was set
+    if package is not None and dest is not None:
+        pkg_data = get_file('%s/%s' % (archive, package['Filename']))
+        if package['SHA256'] != get_sha256(pkg_data):
+            sys.stderr.write(
+                'SHA256 mismatch on %s from %s' % (pkg_name, base_url))
+            sys.exit(1)
+        pkg_path = os.path.join(dest, os.path.basename(package['Filename']))
+        with open(pkg_path, 'wb') as stream:
+            stream.write(pkg_data)
+    return package
+
+
+def extract_files_from_packages(
+        archive, packages, architecture, files, release, dest,
+        grub_format=None):
+    tmp = tempfile.mkdtemp(prefix='maas-images-')
+    for package in packages:
+        package = get_package(archive, package, architecture, release, tmp)
+        pkg_path = os.path.join(tmp, os.path.basename(package['Filename']))
+        if pkg_path is None:
+            sys.stderr.write('%s not found in archives!' % package)
+            sys.exit(1)
+        subprocess.check_output(['dpkg', '-x', pkg_path, tmp])
+
+    if grub_format is None:
+        for i in files:
+            src = "%s/%s" % (tmp, i)
+            if '*' in src or '?' in src:
+                unglobbed_files = glob.glob(src)
+            else:
+                unglobbed_files = [src]
+            for f in unglobbed_files:
+                dest_file = "%s/%s" % (dest, os.path.basename(f))
+                shutil.copyfile(f, dest_file)
+    else:
+        # You can only tell grub to use modules from one directory
+        modules_path = "%s/%s" % (tmp, files[0])
+        modules = []
+        for module_path in glob.glob("%s/*.mod" % modules_path):
+            module_filename = os.path.basename(module_path)
+            module_name, _ = os.path.splitext(module_filename)
+            modules.append(module_name)
+        subprocess.check_output(
+            ['grub-mkimage',
+             '-o', dest,
+             '-O', grub_format,
+             '-d', modules_path,
+             '-c', dest] + modules)
+    shutil.rmtree(tmp)

=== modified file 'meph2/commands/meph2_util.py'
--- meph2/commands/meph2_util.py	2016-06-06 15:49:10 +0000
+++ meph2/commands/meph2_util.py	2016-07-21 22:02:46 +0000
@@ -1,6 +1,7 @@
 #!/usr/bin/python3
 
 import argparse
+import glob
 import copy
 import os
 from functools import partial
@@ -13,6 +14,10 @@
 
 from meph2 import util
 from meph2.url_helper import geturl_text
+from meph2.commands.dpkg import (
+    get_package,
+    extract_files_from_packages,
+)
 
 from simplestreams import (
     contentsource,
@@ -406,10 +411,12 @@
 def load_products(path, product_streams):
     products = {}
     for product_stream in product_streams:
-        with contentsource.UrlContentSource(
-                os.path.join(path, product_stream)) as tcs:
-            product_listing = sutil.load_content(tcs.read())
-        products.update(product_listing['products'])
+        product_stream_path = os.path.join(path, product_stream)
+        if os.path.exists(product_stream_path):
+            with contentsource.UrlContentSource(
+                    product_stream_path) as tcs:
+                product_listing = sutil.load_content(tcs.read())
+                products.update(product_listing['products'])
     return products
 
 
@@ -455,25 +462,7 @@
     return 0
 
 
-def main_import(args):
-    cfg_path = os.path.join(
-        os.path.dirname(__file__), "..", "..", "conf", args.import_cfg)
-    if not os.path.exists(cfg_path):
-        if os.path.exists(args.import_cfg):
-            cfg_path = args.import_cfg
-        else:
-            print("Error: Unable to find config file %s" % args.import_cfg)
-            os.exit(1)
-
-    target_product_streams = load_product_streams(args.target)
-    target_products = load_products(args.target, target_product_streams)
-
-    with open(cfg_path) as fp:
-        cfgdata = yaml.load(fp)
-
-    product_tree = util.empty_iid_products(cfgdata['content_id'])
-    product_tree['updated'] = sutil.timestamp()
-    product_tree['datatype'] = 'image-downloads'
+def import_sha256(args, product_tree, cfgdata):
     for (release, release_info) in cfgdata['versions'].items():
         if 'arch' in release_info:
             arch = release_info['arch']
@@ -483,34 +472,40 @@
             os_name = release_info['os']
         else:
             os_name = cfgdata['os']
-
-        product_id = cfgdata['product_id'].format(
-            version=release_info['version'], arch=arch)
-
-        # If the product already exists don't regenerate the image, just copy
-        # its metadata
-        if product_id in target_products:
-            product_tree['products'][product_id] = target_products[product_id]
-            continue
-
-        product_tree['products'][product_id] = {
-            'subarches': 'generic',
-            'label': 'daily',
-            'subarch': 'generic',
-            'arch': arch,
-            'os': os_name,
-            'version': release_info['version'],
-            'release': release,
-            'versions': {},
-            }
         if 'path_version' in release_info:
             path_version = release_info['path_version']
         else:
             path_version = release_info['version']
+        product_id = cfgdata['product_id'].format(
+            version=release_info['version'], arch=arch)
         url = cfgdata['sha256_meta_data_path'].format(version=path_version)
+        images = get_sha256_meta_images(url)
         base_url = os.path.dirname(url)
-        images = get_sha256_meta_images(url)
+
+        if product_tree['products'].get(product_id) is None:
+            print("Creating new product %s" % product_id)
+            product_tree['products'][product_id] = {
+                'subarches': 'generic',
+                'label': 'daily',
+                'subarch': 'generic',
+                'arch': arch,
+                'os': os_name,
+                'version': release_info['version'],
+                'release': release,
+                'versions': {},
+            }
+
         for (image, image_info) in images.items():
+            if (
+                    product_id in product_tree['products'] and
+                    image in product_tree['products'][product_id]['versions']):
+                print(
+                    "Product %s at version %s exists, skipping" % (
+                        product_id, image))
+                continue
+            print(
+                "Downloading and creating %s version %s" % (
+                    (product_id, image)))
             image_path = '/'.join([release, arch, image, 'root-tgz'])
             real_image_path = os.path.join(
                 os.path.realpath(args.target), image_path)
@@ -528,12 +523,131 @@
                         }
                     }
                 }
+
+
+def get_file_info(f):
+    size = 0
+    sha256 = hashlib.sha256()
+    with open(f, 'rb') as f:
+        for chunk in iter(lambda: f.read(2**15), b''):
+            sha256.update(chunk)
+            size += len(chunk)
+    return sha256.hexdigest(), size
+
+
+def import_bootloaders(args, product_tree, cfgdata):
+    for bootloader in cfgdata['bootloaders']:
+        product_id = cfgdata['product_id'].format(
+            bootloader=bootloader['bootloader'])
+        package = get_package(
+            bootloader['archive'], bootloader['packages'][0],
+            bootloader['arch'], bootloader['release'])
+
+        if (
+                product_id in product_tree['products'] and
+                package['Version'] in product_tree['products'][product_id][
+                    'versions']):
+            print(
+                "Product %s at version %s exists, skipping" % (
+                    product_id, package['Version']))
+            continue
+        if product_tree['products'].get(product_id) is None:
+            print("Creating new product %s" % product_id)
+            product_tree['products'][product_id] = {
+                'label': 'daily',
+                'arch': bootloader['arch'],
+                'subarch': 'generic',
+                'subarches': 'generic',
+                'os': 'bootloader',
+                'release': bootloader['bootloader'],
+                'versions': {},
+                }
+        path = os.path.join(
+            'bootloaders', bootloader['bootloader'], bootloader['arch'],
+            package['Version'])
+        dest = os.path.join(args.target, path)
+        os.makedirs(dest)
+        grub_format = bootloader.get('grub_format')
+        if grub_format is not None:
+            dest = os.path.join(dest, bootloader['grub_output'])
+        print(
+            "Downloading and creating %s version %s" % (
+                product_id, package['Version']))
+        extract_files_from_packages(
+            bootloader['archive'], bootloader['packages'],
+            bootloader['arch'], bootloader['files'], bootloader['release'],
+            dest, grub_format)
+        if grub_format is not None:
+            sha256, size = get_file_info(dest)
+            product_tree['products'][product_id]['versions'][
+                package['Version']] = {
+                'items': {
+                    bootloader['grub_output']: {
+                        'ftype': 'bootloader',
+                        'sha256': sha256,
+                        'path': os.path.join(
+                            path, bootloader['grub_output']),
+                        'size': size,
+                        }
+                    }
+                }
+        else:
+            items = {}
+            for i in bootloader['files']:
+                basename = os.path.basename(i)
+                dest_file = os.path.join(dest, basename)
+                if '*' in dest_file or '?' in dest_file:
+                    unglobbed_files = glob.glob(dest_file)
+                else:
+                    unglobbed_files = [dest_file]
+                for f in unglobbed_files:
+                    basename = os.path.basename(f)
+                    sha256, size = get_file_info(f)
+                    items[basename] = {
+                        'ftype': 'bootloader',
+                        'sha256': sha256,
+                        'path': os.path.join(path, basename),
+                        'size': size,
+                    }
+                product_tree['products'][product_id]['versions'][
+                    package['Version']] = {'items': items}
+
+
+def main_import(args):
+    cfg_path = os.path.join(
+        os.path.dirname(__file__), "..", "..", "conf", args.import_cfg)
+    if not os.path.exists(cfg_path):
+        if os.path.exists(args.import_cfg):
+            cfg_path = args.import_cfg
+        else:
+            print("Error: Unable to find config file %s" % args.import_cfg)
+            os.exit(1)
+
+    with open(cfg_path) as fp:
+        cfgdata = yaml.load(fp)
+
+    target_product_stream = os.path.join(
+        'streams', 'v1', cfgdata['content_id'] + '.json')
+
+    product_tree = util.empty_iid_products(cfgdata['content_id'])
+    product_tree['products'] = load_products(
+        args.target, [target_product_stream])
+    product_tree['updated'] = sutil.timestamp()
+    product_tree['datatype'] = 'image-downloads'
+
+    if cfgdata.get('sha256_meta_data_path', None) is not None:
+        import_sha256(args, product_tree, cfgdata)
+    elif cfgdata.get('bootloaders', None) is not None:
+        import_bootloaders(args, product_tree, cfgdata)
+    else:
+        sys.stderr.write('Unsupported import yaml!')
+        sys.exit(1)
+
     md_d = os.path.join(args.target, 'streams', 'v1')
     if not os.path.exists(md_d):
         os.makedirs(md_d)
 
-    product_tree_fn = cfgdata['content_id'] + '.json'
-    with open(os.path.join(md_d, product_tree_fn), 'wb') as fp:
+    with open(os.path.join(args.target, target_product_stream), 'wb') as fp:
         fp.write(util.dump_data(product_tree))
 
     gen_index_and_sign(args.target, not args.no_sign)

=== modified file 'meph2/netinst.py'
--- meph2/netinst.py	2016-06-07 17:30:20 +0000
+++ meph2/netinst.py	2016-07-21 22:02:46 +0000
@@ -113,7 +113,7 @@
     "dtb": re.compile(r".dtb$").search,
 }
 
-IGNORED_INITRD_FLAVORS =  ('xen', 'cdrom', 'gtk', 'hd-media')
+IGNORED_INITRD_FLAVORS = ('xen', 'cdrom', 'gtk', 'hd-media')
 
 # #
 # # Under a path like: MIRROR/precise-updates/main/installer-i386/