Merge lp:~ltrager/maas-images/squashfs into lp:maas-images

[Lee Trager]

If available build root-image.gz based off the upstream SquashFS image and republish the SquashFS image.

When generating Ubuntu images check if the upstream SimpleStream has published a SquashFS image. If so build root-image.gz based off the SquashFS image, instead of the tar.gz. The downloaded SquashFS image has its SHA256 verified and republished, with the same name as upstream, in the new stream alongside root-image.gz.

This won't be pushed into the image creation process for http://images.maas.io/ephemeral-v2/ for another couple of weeks. I need this to add SquashFS support to MAAS and wanted to get the review out of the way.

[Andres Rodriguez (andreserl)]

Why do we need to create a root-image.gz anymore ? I don't think we want to create any images anymore. If there's a squashfs image, then we should use that and not have to create more and more and more stuff.

We are trying to reduce the amount of images and what we have available and what we use, creating a root-image.gz adds more stuff that we dont really need?

review: Needs Information

[Lee Trager (ltrager)]

> Why do we need to create a root-image.gz anymore ? I don't think we want to
> create any images anymore. If there's a squashfs image, then we should use
> that and not have to create more and more and more stuff.
>
> We are trying to reduce the amount of images and what we have available and
> what we use, creating a root-image.gz adds more stuff that we dont really
> need?

root-image.gz is still needed for older releases until the SquashFS work is done. By having both we can update the stream while still developing the feature. Once the SquashFS work is done and has been pushed out we can remove the root-image.gz.

[Lee Trager (ltrager)]

Last night I created an MP which adds support for SquashFS images to MAAS.

https://code.launchpad.net/~ltrager/maas/squashfs/+merge/299264

[lp:~ltrager/maas-images/squashfs]

305. By Scott Moser on 2016-07-11

Disable metadata_csum when creating ext4 filesystems

In yakkety, we have sync'd an e2fsprogs version that supports metadata
checksumming in ext4 and enables the feature by default. Unfortunately,
some Launchpad buildds (specifically those for armhf and one for powerpc)
are running on a kernel that is too old to support this, which means an
unusable ext4 partition is created. To work around this issue, we disable
the metadata_csum option when we create an ext4 filesystem.

The Launchpad team are working to upgrade the offending build hosts, so we
should be able to back this change out before yakkety releases.

[Lee Trager (ltrager)]

I've added a flag(--squashfs) to meph2-cloudimg-sync. By default this branch now has no effect unless the flag is used. When used if a SquashFS image is available from the upstream SimpleStream the SquashFS image will be published alongside the boot kernel and initrd. The di-kernel, di-initrd, and root-image.gz will no longer be included.

I've also removed the code which downloads the squashfs.manifest file as we don't use it in MAAS.

[lp:~ltrager/maas-images/squashfs]

306. By Lee Trager on 2016-07-25

Import and publish bootloaders from a yaml file.

This allows the import subcommand to accept a yaml file containing a list of bootloaders. The latest version of the bootloader is downloaded from the Ubuntu archive. Files can be copied out of the bootloader or they can be processed with grub-mkimage.

This also fixes a bug with the import command where if any version of a product existed no newer version would be imported. This will allow more up-to-date CentOS images to be published.

307. By Lee Trager on 2016-07-27

Remove duplicated geturl function and add version to bootloader product listing

[Andres Rodriguez (andreserl)]

Questions inline.

review: Needs Information

[lp:~ltrager/maas-images/squashfs]

308. By Lee Trager on 2016-07-29

Allow a grub.cfg to be embedded into generated grub image

309. By Lee Trager on 2016-07-29

Allow bootloader files to be renamed when put into the stream

310. By Lee Trager on 2016-07-29

Strip renamed filenames

311. By Scott Moser on 2016-08-01

tox.ini: tweaks

 * recreate: do not recreate tox environments on every 'tox' run.
 * create separate tox environments rather than re-using via
   declaring the 'envdir'
 * flake8: list the directories to run flake8 on so it does
   not attempt to run on .tox/

312. By Scott Moser on 2016-08-01

dpkg.py: fix flake8 complaints

Simply fix flake8 complaints from dpkg.py.

313. By Scott Moser on 2016-08-01

netinst.py: handle identical data published at different paths

the process that builds the d-i netboot data began publishing a
kernel and initramfs at different paths in 20101020ubuntu318.39.
For example, under
   /ubuntu-ports/dists/trusty-updates/main/installer-arm64/\
      20101020ubuntu318.40/images/xenial-generic/netboot/

There are now the same kernel and initrd and kernel at
  xenial-generic/netboot/initrd.gz
  xenial-generic/netboot/ubuntu-installer/arm64/initrd.gz
  xenial-generic/netboot/ubuntu-installer/arm64/linux
  xenial-generic/netboot/vmlinuz

That would cause a Name Collision in mine_md error on these files.

The change to address this is to raise exception only if
a name collision has different data than the existing. Also
we specifically handle 'dtb' as currently the basename (derived
from the url) is used there.

[Lee Trager (ltrager)]

Responded inline

[lp:~ltrager/maas-images/squashfs]

314. By Lee Trager on 2016-08-01

Use /usr/bin/dpkg directly instead of apt_pkg to avoid tox build issues

315. By Lee Trager on 2016-08-01

Remove unneeded comparison

[Andres Rodriguez (andreserl)]

lgtm!

review: Approve

[lp:~ltrager/maas-images/squashfs]

316. By Lee Trager on 2016-08-01

Add --squashfs flag to meph2-cloudimg-sync

meph2-cloudimg-sync now downloads SquashFS images if available from the upstream source. If the --squashfs flag is used meph2-cloudimg-sync will publish the unmodified SquashFS images where available. The --squashfs flag also disables generating the di-kernel and di-initrd.

=== modified file 'bin/maas-cloudimg2eph2'
--- bin/maas-cloudimg2eph2	2016-07-11 15:18:00 +0000
+++ bin/maas-cloudimg2eph2	2016-08-01 19:18:07 +0000
@@ -646,6 +646,15 @@
         mv --force "$timg" "$output" ||
             { error "failed move to $output"; return 1; }
     fi
+
+    if [[ "$src" == *squashfs ]]; then
+        output_dir=$(dirname $output)
+        cp "$src" "$output_dir" || {
+            error "copy SquashFS image from $src -> $output_dir failed"
+            return 1;
+        }
+    fi
+
     ddebug 1 "finished"
     return 0
 }
 
=== modified file 'meph2/commands/cloudimg_sync.py'
--- meph2/commands/cloudimg_sync.py	2016-03-01 16:01:41 +0000
+++ meph2/commands/cloudimg_sync.py	2016-08-01 19:18:07 +0000
@@ -11,7 +11,9 @@
 from meph2.stream import CONTENT_ID, create_version
 
 import argparse
+import glob
 import copy
+import hashlib
 import os
 import sys
 import yaml
@@ -65,7 +67,7 @@
 
 class CloudImg2Meph2Sync(mirrors.BasicMirrorWriter):
     def __init__(self, config, out_d, target, v2config, rebuilds=None,
-                 verbosity=0):
+                 verbosity=0, squashfs=False):
         super(CloudImg2Meph2Sync, self).__init__(config=config)
         if rebuilds is None:
             rebuilds = {}
@@ -74,7 +76,13 @@
         self.target = target
         self.v2config = v2config
         self.filters = self.config.get('filters', [])
-        self.enable_di = self.config.get('enable_di', True)
+        self.squashfs = squashfs
+        if self.squashfs:
+            # As of MAAS 2.0 DI is no longer supported but SquashFS is.
+            # Since the DI won't be used don't generate them.
+            self.enable_di = False
+        else:
+            self.enable_di = self.config.get('enable_di', True)
 
         with open(v2config) as fp:
             cfgdata = yaml.load(fp)
@@ -117,6 +125,25 @@
         self.content_t = my_prods
         return v2_to_cloudimg_products(my_prods, rebuilds=self.rebuilds)
 
+    def _verify_sha256(self, filename, expected_sha256):
+        sha256 = hashlib.sha256()
+        with open(filename, 'rb') as f:
+            while True:
+                data = f.read(2**18)
+                if not data:
+                    break
+                sha256.update(data)
+        if sha256.hexdigest() != expected_sha256:
+            raise ValueError(
+                'Expected SHA256 %s got %s on %s' %
+                (expected_sha256, sha256.hexdigest(), filename))
+
+    def _remove_unused(self, filename):
+        """Remove the specified file if a squashfs image exists."""
+        squashfs_image = os.path.join(os.path.dirname(filename), '*.squashfs')
+        if len(glob.glob(squashfs_image)) > 0 and os.path.exists(filename):
+            os.remove(filename)
+
     def insert_item(self, data, src, target, pedigree, contentsource):
         # create the ephemeral root
 
@@ -138,8 +165,37 @@
 
         for prodname, items in cvret.items():
             for i in items:
-                sutil.products_set(self.content_t, items[i],
-                                   (prodname, vername, i))
+                filename = os.path.join(self.out_d, items[i]['path'])
+                if 'squashfs' in i and not self.squashfs:
+                    # If we're not publishing the SquashFS image but one
+                    # was used to generate root-image.gz delete it.
+                    os.remove(filename)
+                    continue
+                elif i == 'root-image.gz' and self.squashfs:
+                    # If we're publishing the SquashFS image we don't need the
+                    # root-image after its been used to generate kernels
+                    self._remove_unused(filename)
+                    continue
+                elif i == 'manifest' and self.squashfs:
+                    # If we're publishing the SquashFS image we don't need the
+                    # root-image manifest either.
+                    self._remove_unused(filename)
+                    continue
+                if i == 'squashfs':
+                    # Verify upstream SHA256 of SquashFS images and add
+                    # SHA256 and size to our stream.
+                    self._verify_sha256(filename, flat['sha256'])
+                    items[i]['sha256'] = flat['sha256']
+                    items[i]['size'] = int(flat['size'])
+                elif i == 'squashfs.manifest':
+                    manifest_pedigree = pedigree[:2] + ('squashfs.manifest',)
+                    manifest_flat = sutil.products_exdata(
+                        src, manifest_pedigree)
+                    self._verify_sha256(filename, manifest_flat['sha256'])
+                    items[i]['sha256'] = manifest_flat['sha256']
+                    items[i]['size'] = int(manifest_flat['size'])
+                sutil.products_set(
+                    self.content_t, items[i], (prodname, vername, i))
 
     def insert_products(self, path, target, content):
         tree = copy.deepcopy(self.content_t)
@@ -184,7 +240,12 @@
         return True
 
     def filter_item(self, data, src, target, pedigree):
-        if data['ftype'] != "tar.gz":
+        # Only use tar.gz if no SquashFS image is available
+        if data['ftype'] == 'tar.gz':
+            product = src['products'][pedigree[0]]['versions'][pedigree[1]]
+            if 'squashfs' in product['items'].keys():
+                return False
+        elif data['ftype'] != 'squashfs':
             return False
         return filters.filter_item(self.filters, data, src, pedigree)
 
@@ -212,6 +273,9 @@
     parser.add_argument('--verbose', '-v', action='count', default=0)
     parser.add_argument('--log-file', default=sys.stderr,
                         type=argparse.FileType('w'))
+    parser.add_argument('--squashfs', action='store_true', default=False,
+                        help='Download SquashFS root file systems if available'
+                        )
 
     parser.add_argument('output_d')
     parser.add_argument('filters', nargs='*', default=[])
@@ -272,7 +336,8 @@
 
     tmirror = CloudImg2Meph2Sync(config=mirror_config, out_d=args.output_d,
                                  target=args.target, v2config=args.config,
-                                 rebuilds=rebuilds, verbosity=vlevel)
+                                 rebuilds=rebuilds, verbosity=vlevel,
+                                 squashfs=args.squashfs)
 
     tmirror.sync(smirror, initial_path)
 
 
=== modified file 'meph2/commands/meph2_util.py'
--- meph2/commands/meph2_util.py	2016-07-29 22:28:22 +0000
+++ meph2/commands/meph2_util.py	2016-08-01 19:18:07 +0000
@@ -629,7 +629,7 @@
             cfg_path = args.import_cfg
         else:
             print("Error: Unable to find config file %s" % args.import_cfg)
-            os.exit(1)
+            sys.exit(1)
 
     with open(cfg_path) as fp:
         cfgdata = yaml.load(fp)
 
=== modified file 'meph2/stream.py'
--- meph2/stream.py	2016-05-11 18:41:21 +0000
+++ meph2/stream.py	2016-08-01 19:18:07 +0000
@@ -20,6 +20,10 @@
 PATH_FORMATS = {
     'root-image.gz': PATH_COMMON + "%(version_name)s/root-image.gz",
     'manifest': PATH_COMMON + "%(version_name)s/root-image.manifest",
+    'squashfs': PATH_COMMON + "%(version_name)s/%(img_name)s",
+    'squashfs.manifest': (
+        PATH_COMMON + "%(version_name)s/%(img_name)s.manifest"
+    ),
     'boot-dtb': BOOT_COMMON + "/boot-dtb%(suffix)s",
     'boot-kernel': BOOT_COMMON + "/boot-kernel%(suffix)s",
     'boot-initrd': BOOT_COMMON + "/boot-initrd%(suffix)s",
@@ -116,7 +120,8 @@
     newitems = {}
 
     subs = {'release': release, 'arch': arch,
-            'version_name': version_name, 'version': version}
+            'version_name': version_name, 'version': version,
+            'img_name': os.path.basename(img_url)}
 
     rootimg_path = PATH_FORMATS['root-image.gz'] % subs
     manifest_path = PATH_FORMATS['manifest'] % subs
@@ -151,6 +156,8 @@
 
         boot_keys = ['boot-kernel', 'boot-initrd']
         ikeys = boot_keys + ['root-image.gz', 'manifest']
+        if img_url.endswith('.squashfs'):
+            ikeys += ['squashfs']
 
         dtb = kdata.get('dtb')
         if dtb: