Merge lp:~ltrager/maas-images/rename_squashfs into lp:maas-images

[Lee Trager]

Rename any SquashFS image to squashfs and include all versions in the bootloaders version string.

This renames any SquashFS file to squashfs, its filetype. This makes SquashFS images consistent with other filetypes which use their filetype as their filename. While testing I also noticed that the latest UEFI bootloader wasn't being downloaded. This is because the UEFI bootloader pulls from two different packages, shim.signed and grub-efi-amd64-signed. Previously the first version string was being set as the bootloader version, now if there are multiple versions they are combined into one string.

I've regenerated the beta V3 stream at http://162.213.35.187/v3/ and verified all images can deploy with https://code.launchpad.net/~ltrager/maas/bootloaders/+merge/301584

[Andres Rodriguez (andreserl)]

lgtm! Can you please update that this wont break MAAS ?

Thanks.

review: Approve

[lp:~ltrager/maas-images/rename_squashfs]

322. By Lee Trager on 2016-08-09

Store bootloaders in the base directory

[Scott Moser (smoser)]

looking at http://162.213.35.187/v3/ , you've added several things to the top level directory. Previously (v2) we only had "os" things there. Perhaps you could put all the bootloader things under a bootloaders/ dir ?

Is there value in providing all the files in
  http://162.213.35.187/v3/pxe/amd64/3:6.03+dfsg-11ubuntu1/
individually?
There are a *lot* of files there. if they're not useful individually then archiving them to a .tar might make sense.

simplestream versions (in the images streams 20160802) are C locale sorted to indicate newer or older. Are you sure that the version strings that you've used are guaranteed to continually sort newer in the C locale?
Ie,
  Will all possible future versions of '3:6.03+dfsg-11ubuntu1' or '1.66.2+2.02~beta2-36ubuntu3.2&1.18~16.04.1+0.8-0ubuntu2' sort in C locale correctly? It seems like there are cases where they would not (dpkg sorting is not simply C locale string based and the combined dpkg versions are even less likely to guarantee incremented).

The thing that we've done for the 'd-i' kernels was to use the YYYYMMDD formated string of the first time we came upon these files in the archive. Then you're guaranteed to increment with dates.

I'd prefer 'root-squashfs' or 'root-image.squashfs' as the filename for the squash image, but that is a nitpick and your usage of the ftype is not unreasonable. I just find 'squashfs' to not really tell you anything about the file (i have no idea of the contents by the filename).

one inline comment

[lp:~ltrager/maas-images/rename_squashfs]

323. By Lee Trager on 2016-08-11

Limit the amount of PXE files provided, put all bootloaders in bootloaders dir, make bootloader version based on date stamp, store package names and version the bootloader came from

324. By Lee Trager on 2016-08-11

Use rename instead of move

325. By Lee Trager on 2016-08-11

Copy extra version fields during insert

326. By Lee Trager on 2016-08-12

Get all bootloaders from yakkety

327. By Lee Trager on 2016-08-12

Fix tox errors

328. By Lee Trager on 2016-08-12

Preserve other version items during insert

[Lee Trager (ltrager)]

> looking at http://162.213.35.187/v3/ , you've added several things to the top
> level directory. Previously (v2) we only had "os" things there. Perhaps you
> could put all the bootloader things under a bootloaders/ dir ?

I actually originally had that but was trying to conform with what we were doing with the other images. I'll go back to putting everything in a bootloaders directory.

>
> Is there value in providing all the files in
> http://162.213.35.187/v3/pxe/amd64/3:6.03+dfsg-11ubuntu1/
> individually?
> There are a *lot* of files there. if they're not useful individually then
> archiving them to a .tar might make sense.

I limited the amount of PXE files we are publishing to just the ones we need(6).

>
> simplestream versions (in the images streams 20160802) are C locale sorted to
> indicate newer or older. Are you sure that the version strings that you've
> used are guaranteed to continually sort newer in the C locale?
> Ie,
> Will all possible future versions of '3:6.03+dfsg-11ubuntu1' or
> '1.66.2+2.02~beta2-36ubuntu3.2&1.18~16.04.1+0.8-0ubuntu2' sort in C locale
> correctly? It seems like there are cases where they would not (dpkg sorting
> is not simply C locale string based and the combined dpkg versions are even
> less likely to guarantee incremented).
>
> The thing that we've done for the 'd-i' kernels was to use the YYYYMMDD
> formated string of the first time we came upon these files in the archive.
> Then you're guaranteed to increment with dates.

I've changed the verisoning to use the date stamp as you suggested. In order to keep track of which packages are currently in the stream I've added a new item to the version, src_packages. src_packages contains a list of packages, their versions, and the release they were pulled from. This information may become useful to MAAS.

>
> I'd prefer 'root-squashfs' or 'root-image.squashfs' as the filename for the
> squash image, but that is a nitpick and your usage of the ftype is not
> unreasonable. I just find 'squashfs' to not really tell you anything about
> the file (i have no idea of the contents by the filename).
>

I agree that squashfs isn't the most descriptive name which was why I originally kept the upstream name. I'd prefer to either conform to the upstream name or the filetype naming convention currently being used in the stream.

> one inline comment

Fixed.

I've updated http://162.213.35.187/v3/ to reflect the above changes and retested building the stream and deploying with MAAS.

[Scott Moser (smoser)]

2 comments:
a.) as I said above, you should tar or other wise archive the pxe files and possibly even the other files. There is no point in delivering a 'ldlinux.c32' file as a single entity. No user (maas or other) would have an interest in downloading that file, so there is no reason to make it available as a single file.

b.) The way you've added the 'src_packages' is not valid.
Products have Versions
Versions have Items
Any of those can have key/value pairs associated with them, but the values must be be strings or integers, not arbitrary data types.

so instead of
      "src_packages": [
      {
       "name": "grub-ieee1275-bin",
       "release": "yakkety",
       "version": "2.02~beta2-36ubuntu10"
      }
     ]
You should just package_name and package_version on the item if you want to maintain that information. If you have multiple package names, feel free to join them with a comma or something.

c.) Please make sure that a client (like sstream-mirror) can losslessly mirror the content you've provided.

review: Needs Fixing

[lp:~ltrager/maas-images/rename_squashfs]

329. By Lee Trager on 2016-08-15

Stick src_package, src_version, and src_release in each item.

330. By Lee Trager on 2016-08-15

Remove spaces from filename, make ftype sticky

331. By Lee Trager on 2016-08-16

Only take the source package name, don't include the version.

332. By Lee Trager on 2016-08-16

Tar up the bootloader files and include a description

[Lee Trager (ltrager)]

I've updated the branch as we discussed earlier today. I'm looking into removing release and what has to be done in MAAS for that.

Below is the output of com.ubuntu.maas:daily:1:bootloaders-bases-download.json containing two versions of each bootloader.
http://paste.ubuntu.com/23063089/

[lp:~ltrager/maas-images/rename_squashfs]

333. By Lee Trager on 2016-08-17

Don't include release, include the arch

334. By Lee Trager on 2016-08-17

s/btype/bootloader-type/

335. By Lee Trager on 2016-08-17

s/bootloader-bases/bootloader/g

[Lee Trager (ltrager)]

This review is getting a bit conflated with bootloaders. So I can do some SquashFS testing I've merged the original intent of this branch, renaming SquashFS images to 'squashfs' and created a new branch to finish discussing the bootloader changes at https://code.launchpad.net/~ltrager/maas-images/bootloaders_part2/+merge/303241

=== modified file 'conf/bootloaders.yaml'
--- conf/bootloaders.yaml	2016-07-29 22:18:43 +0000
+++ conf/bootloaders.yaml	2016-08-17 15:56:27 +0000
@@ -1,33 +1,39 @@
-product_id: "com.ubuntu.maas.daily:1:bootloaders-bases:{bootloader}"
-content_id: "com.ubuntu.maas:daily:1:bootloaders-bases-download"
+product_id: "com.ubuntu.maas.daily:1:bootloader:{firmware_platform}:{arch}"
+content_id: "com.ubuntu.maas:daily:1:bootloader-download"
 
 bootloaders:
- - bootloader: pxe
+ - firmware-platform: pxe
    packages:
     - pxelinux
     - syslinux-common
-   arch: amd64
+   arch: i386
+   arches: i386,amd64
    archive: http://archive.ubuntu.com/ubuntu
-   release: xenial
+   release: yakkety
+   bootloader-type: pxelinux
    files:
     - /usr/lib/PXELINUX/pxelinux.0
     - /usr/lib/syslinux/modules/bios/*.c32
- - bootloader: uefi
+ - firmware-platform: uefi
    packages:
     - shim-signed
     - grub-efi-amd64-signed
    arch: amd64
+   arches: amd64
    archive: http://archive.ubuntu.com/ubuntu
-   release: xenial
+   release: yakkety
+   bootloader-type: grub-efi-signed
    files:
     - /usr/lib/shim/shim.efi.signed, bootx64.efi
     - /usr/lib/grub/x86_64-efi-signed/grubnetx64.efi.signed, grubx64.efi
- - bootloader: uefi-arm64
+ - firmware-platform: uefi
    packages:
     - grub-efi-arm64-bin
    arch: arm64
+   arches: arm64
    archive: http://ports.ubuntu.com
-   release: xenial
+   release: yakkety
+   bootloader-type: grub-efi
    files:
     - /usr/lib/grub/arm64-efi/
    grub_format: arm64-efi
@@ -41,12 +47,14 @@
       # Failed to load based on MAC address.
       # Load arm64 by default, UEFI only supported by 64-bit
       configfile (pxe)/grub/grub.cfg-default-arm64
- - bootloader: powerkvm
+ - firmware-platform: open-firmware
    packages:
     - grub-ieee1275-bin
    arch: ppc64el
+   arches: ppc64el,ppc64
    archive: http://ports.ubuntu.com
-   release: xenial
+   release: yakkety
+   bootloader-type: grub-ieee1275
    files:
     - /usr/lib/grub/powerpc-ieee1275/
    grub_format: powerpc-ieee1275

=== modified file 'meph2/commands/cloudimg_sync.py'
--- meph2/commands/cloudimg_sync.py	2016-08-08 19:59:18 +0000
+++ meph2/commands/cloudimg_sync.py	2016-08-17 15:56:27 +0000
@@ -160,16 +160,23 @@
         for prodname, items in cvret.items():
             for i in items:
                 filename = os.path.join(self.out_d, items[i]['path'])
-                if 'squashfs' in i and not self.squashfs:
-                    # If we're not publishing the SquashFS image but one
-                    # was used to generate root-image.gz delete it.
-                    if os.path.exists(filename):
-                        os.remove(filename)
-                    continue
+                if i == 'squashfs':
+                    if self.squashfs:
+                        # Verify upstream SHA256 of SquashFS images and add the
+                        # SHA256 and size to our stream.
+                        self._verify_sha256(filename, flat['sha256'])
+                        items[i]['sha256'] = flat['sha256']
+                        items[i]['size'] = int(flat['size'])
+                    else:
+                        # If we're not publishing the SquashFS image but one
+                        # was used to generate root-image.gz delete it.
+                        if os.path.exists(filename):
+                            os.remove(filename)
+                        continue
                 elif i == 'root-image.gz' and self.squashfs:
                     # If we're publishing the SquashFS image we don't need the
-                    # root-image after its been used to generate kernels. Older
-                    # Ubuntu releases (<16.04) don't have SquashFS images
+                    # root-image after its been used to extract the kernels.
+                    # Older Ubuntu releases (<16.04) don't have SquashFS images
                     # published, so only remove if a SquashFS file exists.
                     squashfs_image = os.path.join(
                         os.path.dirname(filename), '*squashfs')
@@ -186,19 +193,6 @@
                         if os.path.exists(filename):
                             os.remove(filename)
                         continue
-                if i == 'squashfs':
-                    # Verify upstream SHA256 of SquashFS images and add
-                    # SHA256 and size to our stream.
-                    self._verify_sha256(filename, flat['sha256'])
-                    items[i]['sha256'] = flat['sha256']
-                    items[i]['size'] = int(flat['size'])
-                elif i == 'squashfs.manifest':
-                    manifest_pedigree = pedigree[:2] + ('squashfs.manifest',)
-                    manifest_flat = sutil.products_exdata(
-                        src, manifest_pedigree)
-                    self._verify_sha256(filename, manifest_flat['sha256'])
-                    items[i]['sha256'] = manifest_flat['sha256']
-                    items[i]['size'] = int(manifest_flat['size'])
                 sutil.products_set(
                     self.content_t, items[i], (prodname, vername, i))
 

=== modified file 'meph2/commands/dpkg.py'
--- meph2/commands/dpkg.py	2016-08-03 13:00:03 +0000
+++ meph2/commands/dpkg.py	2016-08-17 15:56:27 +0000
@@ -6,6 +6,7 @@
 import os
 import re
 import sys
+import tarfile
 import tempfile
 import glob
 
@@ -128,13 +129,93 @@
         pkg_path = os.path.join(dest, os.path.basename(package['Filename']))
         with open(pkg_path, 'wb') as stream:
             stream.write(pkg_data)
+        package['files'] = []
+        output = subprocess.check_output(['dpkg', '-c', pkg_path])
+        for line in output.decode('utf-8').split('\n'):
+            # The file is the last column in the list.
+            file_info = line.split()
+            # Last line is just a newline
+            if len(file_info) == 0:
+                continue
+            # Remove leading '.' if it exists
+            if file_info[-1].startswith('.'):
+                package['files'].append(file_info[-1][1:])
+            else:
+                package['files'].append(file_info[-1])
     return package
 
 
+def get_file_info(f):
+    size = 0
+    sha256 = hashlib.sha256()
+    with open(f, 'rb') as f:
+        for chunk in iter(lambda: f.read(2**15), b''):
+            sha256.update(chunk)
+            size += len(chunk)
+    return sha256.hexdigest(), size
+
+
+def make_item(ftype, src_file, dest_file, stream_path, src_packages):
+    sha256, size = get_file_info(dest_file)
+    for src_package in src_packages:
+        if src_file in src_package['files']:
+            return {
+                'ftype': ftype,
+                'sha256': sha256,
+                'path': stream_path,
+                'size': size,
+                'src_package': src_package['src_package'],
+                'src_version': src_package['src_version'],
+                'src_release': src_package['src_release'],
+            }
+    raise ValueError("%s not found in src_packages" % src_file)
+
+
+def archive_files(items, target):
+    """Archive multiple files from a src_package into archive.tar.xz."""
+    archive_items = {}
+    new_items = {}
+    # Create a mapping of source packages and the files that came from them.
+    for item in items.values():
+        key = "%(src_package)s-%(src_release)-%(src_version)" % item
+        if archive_items.get(key) is None:
+            archive_items[key] = {
+                'src_package': item['src_package'],
+                'src_release': item['src_release'],
+                'src_version': item['src_version'],
+                'files': [item['path']],
+            }
+        else:
+            archive_items[key]['files'].append(item['path'])
+    for item in archive_items.values():
+        stream_path = os.path.join(
+            os.path.dirname(item['files'][0]),
+            '%s.tar.xz' % item['src_package'])
+        full_path = os.path.join(target, stream_path)
+        tar = tarfile.open(full_path, 'w:xz')
+        for f in item['files']:
+            item_full_path = os.path.join(target, f)
+            tar.add(item_full_path, os.path.basename(item_full_path))
+            os.remove(item_full_path)
+        tar.close()
+        sha256, size = get_file_info(full_path)
+        new_items[item['src_package']] = {
+            'ftype': 'archive.tar.xz',
+            'sha256': sha256,
+            'path': stream_path,
+            'size': size,
+            'src_package': item['src_package'],
+            'src_release': item['src_release'],
+            'src_version': item['src_version'],
+        }
+    return new_items
+
+
 def extract_files_from_packages(
-        archive, packages, architecture, files, release, dest,
-        grub_format=None, grub_config=None):
+        archive, packages, architecture, files, release, target, path,
+        grub_format=None, grub_config=None, grub_output=None):
     tmp = tempfile.mkdtemp(prefix='maas-images-')
+    src_packages = []
     for package in packages:
         package = get_package(archive, package, architecture, release, tmp)
         pkg_path = os.path.join(tmp, os.path.basename(package['Filename']))
@@ -142,7 +223,23 @@
             sys.stderr.write('%s not found in archives!' % package)
             sys.exit(1)
         subprocess.check_output(['dpkg', '-x', pkg_path, tmp])
-
+        new_source_package = True
+        for src_package in src_packages:
+            if src_package['src_package'] == package['Source']:
+                new_source_package = False
+                src_package['files'] += package['files']
+        if new_source_package:
+            # Some source packages include the package version in the source
+            # name. Only take the name, not the version.
+            src_package = package['Source'].split(' ')[0]
+            src_packages.append({
+                'src_package': src_package,
+                'src_version': package['Version'],
+                'src_release': release,
+                'files': package['files'],
+            })
+    dest = os.path.join(target, path)
+    items = {}
     if grub_format is None:
         for i in files:
             if '*' in i or '?' in i:
@@ -150,20 +247,38 @@
                 src = "%s/%s" % (tmp, i)
                 unglobbed_files = glob.glob(src)
                 for f in unglobbed_files:
-                    dest_file = "%s/%s" % (dest, os.path.basename(f))
+                    basename = os.path.basename(f)
+                    dest_file = "%s/%s" % (dest, basename)
+                    stream_path = "%s/%s" % (path, basename)
                     shutil.copyfile(f, dest_file)
+                    pkg_file = f[len(tmp):]
+                    if pkg_file.startswith('//'):
+                        pkg_file = pkg_file[1:]
+                    items[basename] = make_item(
+                        'bootloader', pkg_file, dest_file, stream_path,
+                        src_packages)
             elif ',' in i:
                 # Copy the a file from the package using a new name
                 src_file, dest_file = i.split(',')
-                src_file = "%s/%s" % (tmp, src_file.strip())
-                dest_file = "%s/%s" % (dest, dest_file.strip())
-                shutil.copyfile(src_file, dest_file)
+                dest_file = dest_file.strip()
+                full_src_file_path = "%s/%s" % (tmp, src_file.strip())
+                stream_path = "%s/%s" % (path, dest_file)
+                full_dest_file_path = "%s/%s" % (dest, dest_file)
+                shutil.copyfile(full_src_file_path, full_dest_file_path)
+                items[dest_file] = make_item(
+                    'bootloader', src_file, full_dest_file_path, stream_path,
+                    src_packages)
             else:
                 # Straight copy
+                basename = os.path.basename(i)
                 src_file = "%s/%s" % (tmp,  i)
-                dest_file = "%s/%s" % (dest, os.path.basename(src_file))
+                dest_file = "%s/%s" % (dest, basename)
+                stream_path = "%s/%s" % (path, basename)
                 shutil.copyfile(src_file, dest_file)
+                items[basename] = make_item(
+                    'bootloader', i, dest_file, stream_path, src_packages)
     else:
+        dest = os.path.join(dest, grub_output)
         # You can only tell grub to use modules from one directory
         modules_path = "%s/%s" % (tmp, files[0])
         modules = []
@@ -189,4 +304,9 @@
                  '-O', grub_format,
                  '-d', modules_path,
                  ] + modules)
+        basename = os.path.basename(dest)
+        stream_path = "%s/%s" % (path, basename)
+        items[basename] = make_item(
+            'bootloader', files[0], dest, stream_path, src_packages)
     shutil.rmtree(tmp)
+    return archive_files(items, target)

=== modified file 'meph2/commands/meph2_util.py'
--- meph2/commands/meph2_util.py	2016-08-01 19:15:59 +0000
+++ meph2/commands/meph2_util.py	2016-08-17 15:56:27 +0000
@@ -1,7 +1,7 @@
 #!/usr/bin/python3
 
 import argparse
-import glob
+from datetime import datetime
 import copy
 import os
 from functools import partial
@@ -176,9 +176,13 @@
             (pedigree, sutil.products_exdata(
                 src, pedigree, include_top=False,
                 insert_fieldnames=False)),)
+
         return super(BareMirrorWriter, self).insert_item(
             data, src, target, pedigree, contentsource)
 
+    def remove_item(self, data, src, target, pedigree):
+        return
+
     def remove_version(self, data, src, target, pedigree):
         # sync doesnt filter on things to be removed, so
         # we have to do that here.
@@ -187,9 +191,6 @@
 
         self.removed_versions.append(pedigree)
 
-    def remove_item(self, data, src, target, pedigree):
-        return
-
     def insert_products(self, path, target, content):
         # insert_item and insert_products would not be strictly necessary
         # they're here, though, to keep a list of those things appended.
@@ -231,7 +232,9 @@
 
         sutil.products_condense(
             self.tproducts,
-            sticky=['di_version', 'kpackage', 'sha256', 'md5', 'path'])
+            sticky=[
+                'di_version', 'kpackage', 'sha256', 'md5', 'path', 'ftype',
+                'src_package', 'src_version', 'src_release'])
 
         self.tproducts['updated'] = sutil.timestamp()
 
@@ -525,100 +528,90 @@
                 }
 
 
-def get_file_info(f):
-    size = 0
-    sha256 = hashlib.sha256()
-    with open(f, 'rb') as f:
-        for chunk in iter(lambda: f.read(2**15), b''):
-            sha256.update(chunk)
-            size += len(chunk)
-    return sha256.hexdigest(), size
-
-
 def import_bootloaders(args, product_tree, cfgdata):
-    for bootloader in cfgdata['bootloaders']:
+    for firmware_platform in cfgdata['bootloaders']:
         product_id = cfgdata['product_id'].format(
-            bootloader=bootloader['bootloader'])
-        package = get_package(
-            bootloader['archive'], bootloader['packages'][0],
-            bootloader['arch'], bootloader['release'])
-
-        if (
-                product_id in product_tree['products'] and
-                package['Version'] in product_tree['products'][product_id][
-                    'versions']):
+            firmware_platform=firmware_platform['firmware-platform'],
+            arch=firmware_platform['arch'])
+        # Compile a list of the latest packages in the archive this bootloader
+        # pulls files from
+        src_packages = {}
+        for package in firmware_platform['packages']:
+            package_info = get_package(
+                firmware_platform['archive'], package,
+                firmware_platform['arch'], firmware_platform['release'])
+            # Some source packages include the package version in the source
+            # name. Only take the name, not the version.
+            src_package_name = package_info['Source'].split(' ')[0]
+            src_packages[src_package_name] = {
+                'src_version': package_info['Version'],
+                'src_release': firmware_platform['release'],
+                'found': False,
+            }
+        # Check if the bootloader has been built from the latest version of
+        # the packages in the archive
+        if product_id in product_tree['products']:
+            versions = product_tree['products'][product_id]['versions']
+            for data in versions.values():
+                for item in data['items'].values():
+                    src_package = src_packages.get(item['src_package'])
+                    if (
+                            src_package is not None and
+                            src_package['src_version'] == item['src_version']
+                            and
+                            src_package['src_release'] == item['src_release']):
+                        src_packages[item['src_package']]['found'] = True
+        bootloader_uptodate = True
+        for src_package in src_packages.values():
+            if not src_package['found']:
+                bootloader_uptodate = False
+        # Bootloader built from the latest packages already in stream
+        if bootloader_uptodate:
             print(
-                "Product %s at version %s exists, skipping" % (
-                    product_id, package['Version']))
+                "Product %s built from the latest package set, skipping"
+                % product_id)
             continue
+        # Find an unused version
+        today = datetime.utcnow().strftime('%Y%m%d')
+        point = 0
+        while True:
+            version = "%s.%d" % (today, point)
+            products = product_tree['products']
+            if (
+                    product_id not in products or
+                    version not in products[product_id]['versions'].keys()):
+                break
+            point += 1
         if product_tree['products'].get(product_id) is None:
             print("Creating new product %s" % product_id)
             product_tree['products'][product_id] = {
                 'label': 'daily',
-                'arch': bootloader['arch'],
-                'subarch': 'generic',
-                'subarches': 'generic',
+                'arch': firmware_platform['arch'],
+                'arches': firmware_platform['arches'],
                 'os': 'bootloader',
-                'release': bootloader['bootloader'],
+                'bootloader-type': firmware_platform['bootloader-type'],
                 'versions': {},
                 }
         path = os.path.join(
-            'bootloaders', bootloader['bootloader'], bootloader['arch'],
-            package['Version'])
+            'bootloaders', firmware_platform['firmware-platform'],
+            firmware_platform['arch'], version)
         dest = os.path.join(args.target, path)
         os.makedirs(dest)
-        grub_format = bootloader.get('grub_format')
+        grub_format = firmware_platform.get('grub_format')
         if grub_format is not None:
-            dest = os.path.join(dest, bootloader['grub_output'])
+            dest = os.path.join(dest, firmware_platform['grub_output'])
         print(
             "Downloading and creating %s version %s" % (
-                product_id, package['Version']))
-        extract_files_from_packages(
-            bootloader['archive'], bootloader['packages'],
-            bootloader['arch'], bootloader['files'], bootloader['release'],
-            dest, grub_format, bootloader.get('grub_config'))
-        if grub_format is not None:
-            sha256, size = get_file_info(dest)
-            product_tree['products'][product_id]['versions'][
-                package['Version']] = {
-                'items': {
-                    bootloader['grub_output']: {
-                        'ftype': 'bootloader',
-                        'sha256': sha256,
-                        'path': os.path.join(
-                            path, bootloader['grub_output']),
-                        'size': size,
-                        }
-                    }
-                }
-        else:
-            items = {}
-            for i in bootloader['files']:
-                basename = os.path.basename(i)
-                dest_file = os.path.join(dest, basename)
-                if '*' in dest_file or '?' in dest_file:
-                    # Process multiple files copied with a wildcard
-                    unglobbed_files = glob.glob(dest_file)
-                elif ',' in dest_file:
-                    # If we're renaming the file from the package use the new
-                    # name.
-                    _, basename = i.split(',')
-                    basename = basename.strip()
-                    dest_file = os.path.join(dest, basename)
-                    unglobbed_files = [dest_file]
-                else:
-                    unglobbed_files = [dest_file]
-                for f in unglobbed_files:
-                    basename = os.path.basename(f)
-                    sha256, size = get_file_info(f)
-                    items[basename] = {
-                        'ftype': 'bootloader',
-                        'sha256': sha256,
-                        'path': os.path.join(path, basename),
-                        'size': size,
-                    }
-                product_tree['products'][product_id]['versions'][
-                    package['Version']] = {'items': items}
+                product_id, version))
+        items = extract_files_from_packages(
+            firmware_platform['archive'], firmware_platform['packages'],
+            firmware_platform['arch'], firmware_platform['files'],
+            firmware_platform['release'], args.target, path, grub_format,
+            firmware_platform.get('grub_config'),
+            firmware_platform.get('grub_output'))
+        product_tree['products'][product_id]['versions'][version] = {
+            'items': items
+        }
 
 
 def main_import(args):

=== modified file 'meph2/stream.py'
--- meph2/stream.py	2016-07-22 15:31:57 +0000
+++ meph2/stream.py	2016-08-17 15:56:27 +0000
@@ -20,10 +20,7 @@
 PATH_FORMATS = {
     'root-image.gz': PATH_COMMON + "%(version_name)s/root-image.gz",
     'manifest': PATH_COMMON + "%(version_name)s/root-image.manifest",
-    'squashfs': PATH_COMMON + "%(version_name)s/%(img_name)s",
-    'squashfs.manifest': (
-        PATH_COMMON + "%(version_name)s/%(img_name)s.manifest"
-    ),
+    'squashfs': PATH_COMMON + "%(version_name)s/squashfs",
     'boot-dtb': BOOT_COMMON + "/boot-dtb%(suffix)s",
     'boot-kernel': BOOT_COMMON + "/boot-kernel%(suffix)s",
     'boot-initrd': BOOT_COMMON + "/boot-initrd%(suffix)s",
@@ -120,8 +117,7 @@
     newitems = {}
 
     subs = {'release': release, 'arch': arch,
-            'version_name': version_name, 'version': version,
-            'img_name': os.path.basename(img_url)}
+            'version_name': version_name, 'version': version}
 
     rootimg_path = PATH_FORMATS['root-image.gz'] % subs
     manifest_path = PATH_FORMATS['manifest'] % subs
@@ -243,6 +239,13 @@
         subprocess.check_call(gencmd)
         LOG.info("finished: %s" % gencmd)
 
+    # If we're downloading a SquashFS file rename it to its filetype
+    if img_url.endswith('squashfs'):
+        base_dir = os.path.join(out_d, release, arch, version_name)
+        src_squash = os.path.join(base_dir, os.path.basename(img_url))
+        dst_squash = os.path.join(base_dir, 'squashfs')
+        os.rename(src_squash, dst_squash)
+
     # get checksum and size of new files created
     file_info = {}
     for path in newpaths: