Merge lp:~ltrager/maas-images/kernels into lp:maas-images

[Lee Trager]

Add support for the new kernel naming format. While the new kernel metapackages haven't been published yet this publishes the existing linux-generic, and linux-image-lowlatency-lts-xenial images using the hwe-16.04 and hwe-16.04-lowlatency names. As the new kernels become available I'll add the hwe-16.04-edge, hwe-16.04-lowlatency-edge, hwe-rolling, hwe-rolling-edge, hwe-rolling-lowlatency, and hwe-rolling-lowlatency-edge kernels. Precise and Trusty still use the old names.

Because the V2 stream can't use the new kernel names I had to create a new config file. As the new kernel names a much longer then the old names I converted the yaml arrays from using [] to newlines with -. I only did this for Xenial and Yakkety as they are the only releases that use the new kernels. If we want to make the config uniform I can do the same for Precise and Trusty. I also converted the --squash flag into a config option.

[Blake Rouse (blake-rouse)]

Looks good.

review: Approve

=== added file 'conf/meph-v3.yaml'
--- conf/meph-v3.yaml	1970-01-01 00:00:00 +0000
+++ conf/meph-v3.yaml	2016-08-17 00:25:46 +0000
@@ -0,0 +1,408 @@
+# kernel entries are a list of
+#   kernel-release, arch, primary-subarch, flavor, kernel-pkgname, subarches
+# we use the anchors to make subarches more compact
+#
+# builtin_kernel is "builtin" kernel.
+# default is 'linux-generic'
+#
+# kernel arrays are
+#   krel, karch, primary-subarch, flavor, kernel-package, [subarches], {khdata}
+#
+# 'khdata' is an optional dictionary used to tell which dtb files to pull or
+# what kernel helper to run on the kernel and initramfs.
+
+# Previous to 16.04 kernels were naming using the format
+# hwe-<first letter of release>. From 16.04 onwards kernels come in the format
+# hwe-<release version>. Because the version has a '.' in it we keep the old
+# naming format for the anchors.
+subarches:
+ - &sa_hwe_p
+   - generic
+   - hwe-p
+ - &sa_hwe_q
+   - generic
+   - hwe-p
+   - hwe-q
+ - &sa_hwe_r
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+ - &sa_hwe_s
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+ - &sa_hwe_t
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+ - &sa_hwe_u
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+ - &sa_hwe_v
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+ - &sa_hwe_w
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+   - hwe-w
+ - &sa_hwe_x
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+   - hwe-w
+   - hwe-16.04
+   - hwe-16.04-lowlatency
+ - &sa_hwe_y
+   - generic
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+   - hwe-w
+   - hwe-16.04
+   - hwe-16.04-lowlatency
+   - hwe-16.10
+ - &arm_hwe_t
+   - generic
+   - highbank
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+ - &arm_hwe_u
+   - generic
+   - highbank
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+ - &arm_hwe_v
+   - generic
+   - highbank
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+ - &arm_hwe_w
+   - generic
+   - highbank
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+   - hwe-w
+ - &arm_hwe_x
+   - generic
+   - highbank
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+   - hwe-w
+   - hwe-16.04
+   - hwe-16.04-lowlatency
+ - &arm_hwe_y
+   - generic
+   - highbank
+   - hwe-p
+   - hwe-q
+   - hwe-r
+   - hwe-s
+   - hwe-t
+   - hwe-u
+   - hwe-v
+   - hwe-w
+   - hwe-16.04
+   - hwe-16.04-lowlatency
+   - hwe-16.10
+
+misc:
+ - &glpae generic-lpae
+ - &xguboot xgene-uboot
+ - &mustang xgene-uboot-mustang
+ - &kh-mustang {suffix: .xgene-mustang, kihelper: kh-xgene,
+               di-format: 'xgene', dtb: 'apm-mustang.dtb'}
+ - &kh-xgene   {suffix: .xgene, kihelper: "kh-xgene", di-format: 'xgene'}
+
+releases:
+ - version: "16.10"
+   release: yakkety
+   kernels:
+    -
+      - yakkety
+      - amd64
+      - hwe-16.10
+      - generic
+      - linux-generic
+      - *sa_hwe_y
+    -
+      - yakkety
+      - arm64
+      - hwe-16.10
+      - generic
+      - linux-generic
+      - *sa_hwe_y
+    - 
+      - yakkety
+      - arm64
+      - *mustang
+      - generic
+      - linux-generic
+      - [*mustang]
+      - *kh-mustang
+    -
+      - yakkety
+      - arm64
+      - *xguboot
+      - generic
+      - linux-generic
+      - [*xguboot]
+      - *kh-xgene
+    -
+      - yakkety
+      - armhf
+      - *glpae
+      - *glpae
+      - linux-generic-lpae
+      - [*glpae]
+    -
+      - yakkety
+      - armhf
+      - hwe-16.10
+      - generic
+      - linux-generic
+      - *arm_hwe_y
+    -
+      - yakkety
+      - i386
+      - hwe-16.10
+      - generic
+      - linux-generic
+      - *sa_hwe_y
+    -
+      - yakkety
+      - ppc64el
+      - hwe-16.10
+      - generic
+      - linux-generic
+      - *sa_hwe_y
+    -
+      - yakkety
+      - s390x
+      - hwe-16.10
+      - generic
+      - linux-generic
+      - *sa_hwe_y
+ - version: "16.04"
+   release: xenial
+   kernels:
+    -
+      - xenial
+      - amd64
+      - hwe-16.04
+      - generic
+      - linux-generic
+      - *sa_hwe_x
+    -
+      - xenial
+      - amd64
+      - hwe-16.04-lowlatency
+      - lowlatency
+      - linux-image-lowlatency-lts-xenial
+      - *sa_hwe_x
+    -
+      - xenial
+      - arm64
+      - hwe-16.04
+      - generic
+      - linux-generic
+      - *sa_hwe_x
+    -
+      - xenial
+      - arm64
+      - hwe-16.04-lowlatency
+      - lowlatency
+      - linux-image-lowlatency-lts-xenial
+      - *sa_hwe_x
+    -
+      - xenial
+      - arm64
+      - *mustang
+      - generic
+      - linux-generic
+      - [*mustang]
+      - *kh-mustang
+    -
+      - xenial
+      - arm64
+      - *xguboot
+      - generic
+      - linux-generic
+      - [*xguboot]
+      - *kh-xgene
+    -
+      - xenial
+      - armhf
+      - *glpae
+      - *glpae
+      - linux-generic-lpae
+      - [*glpae]
+    -
+      - xenial
+      - armhf
+      - hwe-16.04
+      - generic
+      - linux-generic
+      - *arm_hwe_x
+    -
+      - xenial
+      - armhf
+      - hwe-16.04-lowlatency
+      - lowlatency
+      - linux-image-lowlatency-lts-xenial
+      - *arm_hwe_x
+    -
+      - xenial
+      - i386
+      - hwe-16.04
+      - generic
+      - linux-generic
+      - *sa_hwe_x
+    -
+      - xenial
+      - i386
+      - hwe-16.04-lowlatency
+      - lowlatency
+      - linux-image-lowlatency-lts-xenial
+      - *sa_hwe_x
+    -
+      - xenial
+      - ppc64el
+      - hwe-16.04
+      - generic
+      - linux-generic
+      - *sa_hwe_x
+    -
+      - xenial
+      - ppc64el
+      - hwe-16.04-lowlatency
+      - lowlatency
+      - linux-image-lowlatency-lts-xenial
+      - *sa_hwe_x
+    -
+      - xenial
+      - s390x
+      - hwe-16.04
+      - generic
+      - linux-generic
+      - *sa_hwe_x
+    -
+      - xenial
+      - s390x
+      - hwe-16.04-lowlatency
+      - lowlatency
+      - linux-image-lowlatency-lts-xenial
+      - *sa_hwe_x
+ - version: "14.04"
+   builtin_kernel: linux-generic
+   release: trusty
+   kernels:
+    - [trusty,  armhf,   hwe-t,    generic,   linux-generic,            *arm_hwe_t]
+    - [trusty,  armhf,   *glpae,   *glpae,    linux-generic-lpae,       [*glpae]]
+    - [trusty,  armhf,   keystone, keystone,  linux-keystone,           [keystone]]
+    - [trusty,  i386,    hwe-t,    generic,   linux-generic,            *sa_hwe_t]
+    - [trusty,  amd64,   hwe-t,    generic,   linux-generic,            *sa_hwe_t]
+    - [trusty,  ppc64el, hwe-t,    generic,   linux-generic,            *sa_hwe_t]
+    - [trusty,  arm64,   hwe-t,    generic,   linux-generic,            *sa_hwe_t]
+    - [trusty,  arm64,   *xguboot, generic,   linux-generic,            [*xguboot], *kh-xgene]
+    - [trusty,  arm64,   *mustang, generic,   linux-generic,            [*mustang], *kh-mustang]
+    - [utopic,  amd64,   hwe-u,    generic,  linux-generic-lts-utopic,  *sa_hwe_u]
+    - [utopic,  arm64,   hwe-u,    generic,  linux-generic-lts-utopic,  *sa_hwe_u]
+    - [utopic,  armhf,   hwe-u,    generic,  linux-generic-lts-utopic,  *sa_hwe_u]
+    - [utopic,  i386,    hwe-u,    generic,  linux-generic-lts-utopic,  *sa_hwe_u]
+    - [utopic,  ppc64el, hwe-u,    generic,  linux-generic-lts-utopic,  *sa_hwe_u]
+    - [vivid,   amd64,   hwe-v,    generic,  linux-generic-lts-vivid,   *sa_hwe_v]
+    - [vivid,   arm64,   hwe-v,    generic,  linux-generic-lts-vivid,   *sa_hwe_v]
+    - [vivid,   armhf,   hwe-v,    generic,  linux-generic-lts-vivid,   *sa_hwe_v]
+    - [vivid,   i386,    hwe-v,    generic,  linux-generic-lts-vivid,   *sa_hwe_v]
+    - [vivid,   ppc64el, hwe-v,    generic,  linux-generic-lts-vivid,   *sa_hwe_v]
+    - [wily,    amd64,   hwe-w,    generic,  linux-generic-lts-wily,    *sa_hwe_w]
+    - [wily,    arm64,   hwe-w,    generic,  linux-generic-lts-wily,    *sa_hwe_w]
+    - [wily,    armhf,   hwe-w,    generic,  linux-generic-lts-wily,    *sa_hwe_w]
+    - [wily,    i386,    hwe-w,    generic,  linux-generic-lts-wily,    *sa_hwe_w]
+    - [wily,    ppc64el, hwe-w,    generic,  linux-generic-lts-wily,    *sa_hwe_w]
+    - [xenial,  amd64,   hwe-x,    generic,  linux-generic-lts-xenial,  *sa_hwe_x]
+    - [xenial,  arm64,   hwe-x,    generic,  linux-generic-lts-xenial,  *sa_hwe_x]
+    - [xenial,  armhf,   hwe-x,    generic,  linux-generic-lts-xenial,  *sa_hwe_x]
+    - [xenial,  i386,    hwe-x,    generic,  linux-generic-lts-xenial,  *sa_hwe_x]
+    - [xenial,  ppc64el, hwe-x,    generic,  linux-generic-lts-xenial,  *sa_hwe_x]
+ - version: "12.04"
+   release: precise
+   builtin_kernel: {armhf: linux-highbank, default: 'linux-generic'}
+   kernels:
+    - [precise, i386,    hwe-p,    generic,  linux-generic,             *sa_hwe_p]
+    - [precise, amd64,   hwe-p,    generic,  linux-generic,             *sa_hwe_p]
+    - [precise, armhf,   highbank, highbank, linux-highbank,            [highbank]]
+    - [quantal, i386,    hwe-q,    generic,  linux-generic-lts-quantal, *sa_hwe_q]
+    - [quantal, amd64,   hwe-q,    generic,  linux-generic-lts-quantal, *sa_hwe_q]
+    - [raring,  i386,    hwe-r,    generic,  linux-generic-lts-raring,  *sa_hwe_r]
+    - [raring,  amd64,   hwe-r,    generic,  linux-generic-lts-raring,  *sa_hwe_r]
+    - [saucy,   i386,    hwe-s,    generic,  linux-generic-lts-saucy,   *sa_hwe_s]
+    - [saucy,   amd64,   hwe-s,    generic,  linux-generic-lts-saucy,   *sa_hwe_s]
+    - [trusty,  i386,    hwe-t,    generic,  linux-generic-lts-trusty,  *sa_hwe_t]
+    - [trusty,  amd64,   hwe-t,    generic,  linux-generic-lts-trusty,  *sa_hwe_t]
+
+# enable_proposed: True
+
+# If available pull and publish SquashFS images from upstream
+squashfs: True
+
+## Notes:
+##  * there are no hwe installer kernels for armhf
+##    nothing named 'saucy-highbank' at
+##     http://ports.ubuntu.com/ubuntu-ports/dists/precise-updates/main/installer-armhf/current/images/MD5SUMS
+##    so for now, we just don't have a stream for that.
+##  * kernel meta package in all cases here is linux-image-<flavor>
 
=== modified file 'meph2/commands/cloudimg_sync.py'
--- meph2/commands/cloudimg_sync.py	2016-08-17 00:25:46 +0000
+++ meph2/commands/cloudimg_sync.py	2016-08-17 00:25:46 +0000
@@ -66,8 +66,8 @@
 
 
 class CloudImg2Meph2Sync(mirrors.BasicMirrorWriter):
-    def __init__(self, config, out_d, target, v2config, rebuilds=None,
-                 verbosity=0, squashfs=False):
+    def __init__(
+            self, config, out_d, target, v2config, rebuilds=None, verbosity=0):
         super(CloudImg2Meph2Sync, self).__init__(config=config)
         if rebuilds is None:
             rebuilds = {}
@@ -76,17 +76,18 @@
         self.target = target
         self.v2config = v2config
         self.filters = self.config.get('filters', [])
-        self.squashfs = squashfs
-        if self.squashfs:
+        with open(v2config) as fp:
+            cfgdata = yaml.load(fp)
+        self.cfgdata = cfgdata
+        if self.cfgdata.get('squashfs'):
+            self.squashfs = True
             # As of MAAS 2.0 DI is no longer supported but SquashFS is.
             # Since the DI won't be used don't generate them.
             self.enable_di = False
         else:
+            self.squashfs = False
             self.enable_di = self.config.get('enable_di', True)
 
-        with open(v2config) as fp:
-            cfgdata = yaml.load(fp)
-        self.cfgdata = cfgdata
         self.releases = []
         for r in [k['release'] for k in cfgdata['releases']]:
             if r not in ubuntu_info.SUPPORTED:
@@ -272,9 +273,6 @@
     parser.add_argument('--verbose', '-v', action='count', default=0)
     parser.add_argument('--log-file', default=sys.stderr,
                         type=argparse.FileType('w'))
-    parser.add_argument('--squashfs', action='store_true', default=False,
-                        help='Download SquashFS root file systems if available'
-                        )
 
     parser.add_argument('output_d')
     parser.add_argument('filters', nargs='*', default=[])
@@ -335,8 +333,7 @@
 
     tmirror = CloudImg2Meph2Sync(config=mirror_config, out_d=args.output_d,
                                  target=args.target, v2config=args.config,
-                                 rebuilds=rebuilds, verbosity=vlevel,
-                                 squashfs=args.squashfs)
+                                 rebuilds=rebuilds, verbosity=vlevel)
 
     tmirror.sync(smirror, initial_path)