Merge lp:~smoser/maas-images/lp1626810 into lp:maas-images

[Scott Moser]

Do not build products for architectures not listed in config.

This change does 2 things:
 a.) enables 'powerpc' to actually build
 b.) makes meph2-build and cloud-image sync not build products for architectures not listed in the config.

The path for 'b' is to write empty products data and to LOG.warn and
write to stderr a message saying how to support.

Also there is a drive by fix here to make 'insert --no-sign' work respect
the --no-sign argument.

[Andres Rodriguez (andreserl)]

While the change looks good there's one thing:

We shouldn't be providing MAAS images for an architecture we don't even support.

[Andres Rodriguez (andreserl)]

forgot the approve.

[lp:~smoser/maas-images/lp1626810]

331. By Scott Moser on 2016-09-23

find supported arch list from the config file.

[Scott Moser (smoser)]

Note... this will create an empty products tree.
I'm not entirely certain that all other tools will handle that
empty products tree. Other tools may need to be updated to address this.

Example:

$ ./bin/meph2-build powerpc yakkety 20101001 /tmp/foo.img out.d
arch 'powerpc' is not supported for release 'yakkety'.
Release has architectures: {'ppc64el', 'arm64', 's390x', 'i386', 'amd64', 'armhf'}.
To support, add kernel info to config.

$ for f in $(find out.d -type f); do echo == $f ==; cat $f; done
== out.d/streams/v1/com.ubuntu.maas:daily:v2:download.json ==
{
 "content_id": "com.ubuntu.maas:daily:v2:download",
 "datatype": "image-downloads",
 "format": "products:1.0",
 "products": {},
 "updated": "Fri, 23 Sep 2016 16:14:25 +0000"
}
== out.d/streams/v1/index.json ==
{
 "format": "index:1.0",
 "index": {
  "com.ubuntu.maas:daily:v2:download": {
   "datatype": "image-downloads",
   "format": "products:1.0",
   "path": "streams/v1/com.ubuntu.maas:daily:v2:download.json",
   "products": [],
   "updated": "Fri, 23 Sep 2016 16:14:25 +0000"
  }
 },
 "updated": "Fri, 23 Sep 2016 16:14:25 +0000"
}

[lp:~smoser/maas-images/lp1626810]

332. By Scott Moser on 2016-09-23

drive by fix, in 'insert' respect --no-sign.

[Scott Moser (smoser)]

This is actually the right result... the user asked us to build all relevant products, and we built none. How better to indicate that than to build an empty list that can be parsed?

[Scott Moser (smoser)]

I've verified at least that 'insert' into an empty dir is fine. i'm guessing it will be fine on other data too:

$ rm -Rf out.d; ./bin/meph2-build powerpc yakkety 20101001 /tmp/foo.img out.d
arch 'powerpc' is not supported for release 'yakkety'.
Release has architectures: {'amd64', 's390x', 'ppc64el', 'armhf', 'arm64', 'i386'}.
To support, add kernel info to config.

$ rm -Rf joined.d; ./bin/meph2-util insert --no-sign out.d/streams/v1/index.json joined.d

[Robert C Jennings (rcj)]

smoser, this looks like a fine solution. +1

[lp:~smoser/maas-images/lp1626810]

333. By Scott Moser on 2016-09-23

fix the case statement. qemu_arch here is now 'ppc'.

334. By Scott Moser on 2016-09-23

mark fixes 1626810

[Blake Rouse (blake-rouse)]

Sounds good.

review: Approve

=== modified file 'bin/maas-cloudimg2ephemeral'
--- bin/maas-cloudimg2ephemeral	2016-09-22 16:38:08 +0000
+++ bin/maas-cloudimg2ephemeral	2016-09-23 16:33:33 +0000
@@ -492,12 +492,14 @@
         arm64) qemu_arch="aarch64";;
         arm*) qemu_arch="arm";;
         ppc64el) qemu_arch="ppc64le";;
+        powerpc) qemu_arch="ppc";;
         *) qemu_arch="$debian_arch";;
     esac
 
     local cross=true
     case "$host_arch:$qemu_arch" in
         $host_arch:$host_arch) cross=false;;
+        ppc64:ppc) cross=false;;
         i?86:i386) cross=false;;
         x86_64:i386) cross=false;;
         aarm64:arm64) cross=false;;
 
=== modified file 'meph2/commands/build_image.py'
--- meph2/commands/build_image.py	2016-09-14 06:58:09 +0000
+++ meph2/commands/build_image.py	2016-09-23 16:33:33 +0000
@@ -24,7 +24,7 @@
             sutil.products_set(prod_tree, items[i],
                                (prodname, version_name, i))
 
-    sutil.products_prune(prod_tree)
+    sutil.products_prune(prod_tree, preserve_empty_products=True)
     sutil.products_condense(prod_tree, sticky=['di_version', 'kpackage'])
 
     tsnow = sutil.timestamp()
 
=== modified file 'meph2/commands/cloudimg_sync.py'
--- meph2/commands/cloudimg_sync.py	2016-09-14 06:58:09 +0000
+++ meph2/commands/cloudimg_sync.py	2016-09-23 16:33:33 +0000
@@ -150,7 +150,7 @@
 
     def insert_products(self, path, target, content):
         tree = copy.deepcopy(self.content_t)
-        sutil.products_prune(tree)
+        sutil.products_prune(tree, preserve_empty_products=True)
         # stop these items from copying up when we call condense
         sutil.products_condense(tree,
                                 sticky=['di_version', 'kpackage'])
 
=== modified file 'meph2/commands/meph2_util.py'
--- meph2/commands/meph2_util.py	2016-08-23 06:08:11 +0000
+++ meph2/commands/meph2_util.py	2016-09-23 16:33:33 +0000
@@ -461,7 +461,7 @@
     tmirror = InsertBareMirrorWriter(config=mirror_config, objectstore=tstore)
     tmirror.sync(smirror, src_path)
 
-    gen_index_and_sign(args.target)
+    gen_index_and_sign(args.target, sign=not args.no_sign)
     return 0
 
 
 
=== modified file 'meph2/stream.py'
--- meph2/stream.py	2016-09-22 16:51:28 +0000
+++ meph2/stream.py	2016-09-23 16:33:33 +0000
@@ -6,6 +6,7 @@
 import copy
 import os
 import subprocess
+import sys
 import yaml
 
 from simplestreams.log import LOG
@@ -32,6 +33,27 @@
                  'root-tar', 'squashfs-image']
 
 
+def read_kdata(info, ret=list):
+    # read a kernel data list and return it as a list or a dict.
+
+    # copy it for our modification.
+    info = list(info)
+
+    # 7th field is optional in kernel lines in config data
+    # so fill it with empty dictionary if not present.
+    if len(info) == 6:
+        info.append({})
+
+    names = ("krel", "arch", "subarch", "flavor", "kpkg",
+             "subarches", "kdata")
+    if ret == list:
+        return info
+    elif ret == dict:
+        return dict(zip(names, info))
+    else:
+        raise ValueError("Unexpected input '%s'" % ret)
+
+
 def create_version(arch, release, version_name, img_url, out_d,
                    include_di=True, cfgdata=None, common_tags=None,
                    verbosity=0, img_format=None):
@@ -75,6 +97,17 @@
                                  release)
             rdata = r
 
+    arches = set([read_kdata(i, dict)['arch'] for i in rdata['kernels']])
+    if arch not in arches:
+        msg = (
+            "arch '%(arch)s' is not supported for release '%(release)s'.\n"
+            "Release has architectures: %(arches)s.\n"
+            "To support, add kernel info to config." %
+            {'arch': arch, 'release': release, 'arches': arches})
+        LOG.warn(msg)
+        sys.stderr.write(msg + "\n")
+        return {}
+        
     version = rdata['version']
     if isinstance(version, float):
         raise ValueError("release '%s' in config had version as a float (%s) "
@@ -142,11 +175,8 @@
     kdata_defaults = {'suffix': "", 'di-format': "default", 'dtb': ""}
 
     for info in rdata['kernels']:
-        # 7th field is optional in kernel lines in config data
-        # so fill it with empty dictionary if not present.
-        if len(info) == 6:
-            info.append({})
-        (krel, karch, psubarch, flavor, kpkg, subarches, kdata) = info
+        (krel, karch, psubarch, flavor, kpkg, subarches, kdata) = (
+            read_kdata(info))
 
         if karch != arch:
             continue