Merge lp:~ltrager/maas-images/bootloaders_proposed into lp:maas-images

[Lee Trager]

Add --proposed to import command for bootloaders.

[Andres Rodriguez (andreserl)]

lgtm!

review: Approve

[Andres Rodriguez (andreserl)]

Line 19/20 you are changing a set to a list. Revert that back and lets do minimum changes.

review: Needs Fixing

[Scott Moser (smoser)]

it looks fine.

i'd appreciate a change to kwargs for those function calls.
just easier to read.

[lp:~ltrager/maas-images/bootloaders_proposed]

386. By Scott Moser on 2018-01-04

Add zfs module and dependencies to generated initramfs.

In order for curtin to use zfs, it has to have access to the modules.
Add 'zfs' and its dependencies to the initramfs.

Note that the user-space tools (zfsutils-linux) are not currently
a dependency of ubuntu-server and are thus not in the squashfs images.
That means that for curtin to use zfs it has to install the package.
The package handles loading zfs module.

387. By Scott Moser on 2018-01-18

Stop grub scripts from running when updating an image.

An upgrade that brought a new kernel would cause grub's kernel
hook scripts to run. Those would fail to run and cause build failure
(at least in trusty). This is similar to code that is currently
in the kpack-from-image, but handles putting things back more elegantly.
(kpack-from-image doesn't have to put things back).

I found this when running with '--proposed' and trusty.

388. By Scott Moser on 2018-02-01

Cleanups around use of --proposed.

The --proposed flag was not getting all the way through to
kpack-from-image. This modification just gets it set all the way through.
It isn't strictly needed, because the image that it runs in will have
proposed enabled already, but passing it through makes it more obvious.

In kpack-from-image, use the same .list file name that is used in
maas-cloudimg2eph and also enable universe and multiverse as are done there.

Also some cleanups:
 - dist_upgrade was set to 'false' by default, but then enabled to either
   'true' or '1'. Just be consistent and set to true to avoid confusion.
 - fix a debug message that reported about ppa when it should have reported
   about proposed.

[Scott Moser (smoser)]

feel free to pull this in.

review: Approve

[lp:~ltrager/maas-images/bootloaders_proposed]

389. By Scott Moser on 2018-02-01

precise: remove reference to non-existing package linux-signed-generic.

Apparently there was no signed generic kernel for precise GA.
Specifically, there is no:
  linux-signed-generic
but there *is*:
  linux-signed-generic-lts-quantal
  linux-signed-generic-lts-raring
  linux-signed-generic-lts-saucy
  linux-signed-generic-lts-trusty

390. By Scott Moser on 2018-02-06

Always create squashfs image if configured.

For precise in the v3 stream there was no 'squashfs' image created.
This was enabled only if input to the build was '.tar.gz' or 'squashfs'.

As we need a squashfs image for installing precise with maas 2.3+,
we modify the code here to build that image.

Note that this squashfs image is known to not boot with rooturl
as the other squashfs images produced do.

391. By Scott Moser on 2018-02-06

Do not install cloud-initramfs-dyn-netconf in releases newer than xenial.

The cloud-initramfs-dyn-netconf was being added to the maas image.
It was only absolutely necessary in <xenial, but is left as present
in xenial as images still use the /etc/network/interfaces link.
See change="symlink ENI for cloud-initramfs-dyn-netconf".

It is not needed in or used without ifupdown, and ifupdown is not present
in artful or bionic.

392. By Scott Moser on 2018-02-12

Re-add cloud-initramfs-dyn-netconf. Needed by maas for BOOTIF in ip=.

The previous commit to trunk dropped cloud-initramfs-dyn-netconf from
the maas image in releases newer than xenial.

That caused errors in MAAS. cloud-initramfs-dyn-netconf does 2 things:
 a.) writes /run/network/dynamic-interfaces
     this is a ENI style file with data retrived from ipconfig
 b.) convert 'BOOTIF' string in the value of IP
   Example:
     ip=::::foobar:BOOTIF BOOTIF=01-00-22-68-10-c1-e6
   will be effectively rendered as:
     ip=::::foobar:eth0

'b' is still required by maas.

Note: in maas v3 images, the squashfs image is re-used pristine
from cloud-images. Because of this, the installation into the image
affects initramfs creation, but does not affect the image. That is
correct in that this package is not needed in the image but only
in the initramfs.

393. By Lee Trager on 2018-04-09

Add hpilo kernel module to ephemeral initrd

394. By Lee Trager on 2018-04-09

Add --proposed for bootloaders

Unmerged revisions

=== modified file 'meph2/commands/dpkg.py'
--- meph2/commands/dpkg.py	2016-08-24 20:40:23 +0000
+++ meph2/commands/dpkg.py	2018-04-09 19:54:24 +0000
@@ -102,7 +102,9 @@
     return ret == 0
 
 
-def get_package(archive, pkg_name, architecture, release=None, dest=None):
+def get_package(
+        archive, pkg_name, architecture, release=None, dest=None,
+        proposed=False):
     """Look through the archives for package metadata. If a dest is given
     download the package.
 
@@ -112,7 +114,10 @@
     release = get_distro_release() if release is None else release
     package = None
     # Find the latest version of the package
-    for dist in ('%s-updates' % release, '%s-security' % release, release):
+    dists = ('%s-updates' % release, '%s-security' % release, release)
+    if proposed:
+        dists = ('%s-proposed' % release,) + dists
+    for dist in dists:
         base_url = '%s/dists/%s' % (archive, dist)
         packages = get_packages(base_url, architecture, pkg_name)
         if pkg_name in packages:
@@ -218,11 +223,12 @@
 
 def extract_files_from_packages(
         archive, packages, architecture, files, release, target, path,
-        grub_format=None, grub_config=None, grub_output=None):
+        grub_format=None, grub_config=None, grub_output=None, proposed=False):
     tmp = tempfile.mkdtemp(prefix='maas-images-')
     src_packages = []
     for package in packages:
-        package = get_package(archive, package, architecture, release, tmp)
+        package = get_package(
+            archive, package, architecture, release, tmp, proposed=proposed)
         pkg_path = os.path.join(tmp, os.path.basename(package['Filename']))
         if pkg_path is None:
             sys.stderr.write('%s not found in archives!' % package)

=== modified file 'meph2/commands/mimport.py'
--- meph2/commands/mimport.py	2017-05-02 07:31:59 +0000
+++ meph2/commands/mimport.py	2018-04-09 19:54:24 +0000
@@ -97,8 +97,9 @@
         src_packages = {}
         for package in firmware_platform['packages']:
             package_info = get_package(
-                firmware_platform['archive'], package,
-                firmware_platform['arch'], firmware_platform['release'])
+                archive=firmware_platform['archive'], pkg_name=package,
+                architecture=firmware_platform['arch'],
+                release=firmware_platform['release'], proposed=args.proposed)
             # Some source packages include the package version in the source
             # name. Only take the name, not the version.
             src_package_name = package_info['Source'].split(' ')[0]
@@ -167,7 +168,7 @@
             firmware_platform['arch'], firmware_platform['files'],
             firmware_platform['release'], args.target, path, grub_format,
             firmware_platform.get('grub_config'),
-            firmware_platform.get('grub_output'))
+            firmware_platform.get('grub_output'), args.proposed)
         product_tree['products'][product_id]['versions'][version] = {
             'items': items
         }
@@ -308,6 +309,9 @@
             args = [args]
         parser.add_argument(*args, **kwargs)
     parser.set_defaults(action=main_import)
+    parser.add_argument(
+        '--proposed', help='Pull bootloaders from proposed',
+        action='store_true', default=False)
 
     args = parser.parse_args()
     if not getattr(args, 'action', None):