Merge lp:~ltrager/maas-images/centos_storage_curthooks into lp:maas-images

[Lee Trager]

Update CentOS 7 curthooks to use internal Curtin storage support.

Curtin now supports deploying custom storage on CentOS 7+. Hooks are no
longer required for CentOS but they are still included for backwards
compatibility. If CentOS is deployed with MAAS 2.5+ and a supported version of Curtin the hooks will simply call back into Curtin to configure storage.
If an older version of MAAS or Curtin is used the old hooks will still run.

Stream with updated CentOS 7 image: http://162.213.35.187/centos-storage/

Testing:
I tested in the CI and configured a BIOS and UEFI machine with 2 disk to have a btrfs filesystem for / on LVM on top of a RAID 1 over 2 disks

MAAS-2.4.1 + curtin-18.1-17-gae48e86f-0ubuntu1~18.04.1(stable) - Flat layout
MAAS-2.4.1 + curtin+centos-storage - Flat layout
MAAS-master + curtin-18.1-17-gae48e86f-0ubuntu1~18.04.1 - Flat layout
MAAS-master + curtin+centos-storage - Custom storage layout deployed!

* curtin+centos-storage - https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/349075

[Lee Trager (ltrager)]

Thanks for the review. I've updated to use the real package name as discussed on IRC. I was also able to confirm custom storage deployment works :)

[lp:~ltrager/maas-images/centos_storage_curthooks]

414. By Lee Trager on 2018-08-23

Cleanup package order

[Andres Rodriguez (andreserl)]

Question inline.

review: Needs Information

[Lee Trager (ltrager)]

We're not supporting custom storage on CentOS 6 so its Curtin hooks remain the same. grub2-efi-x64 is only added to the list of packages added to the CentOS 7 image.

[Andres Rodriguez (andreserl)]

lgtm! But this should land if and only if the hooks are exactly like those in maas-image-builder

review: Approve

[Lee Trager (ltrager)]

Yes they are exactly the same

$ diff -u maas-images/centos_storage_curthooks/curtin/centos7/ maas-image-builder/contrib/centos/centos7/curtin/
$

[lp:~ltrager/maas-images/centos_storage_curthooks]

415. By Lee Trager on 2018-08-29

Don't include grub2-efi-x64

=== modified file 'conf/centos.yaml'
--- conf/centos.yaml	2018-08-16 19:38:27 +0000
+++ conf/centos.yaml	2018-08-29 01:11:32 +0000
@@ -17,6 +17,7 @@
          # cloud-init depends on packages from EPEL
          - epel-release
          - bash-completion
+         - linux-firmware
          # XXX ltrager 2017-07-28 - COPR doesn't provide a URL to the latest
          # package build. Instead it expects you to first add the repo which
          # is what is included in this package.
@@ -28,11 +29,16 @@
          # Without it cloud-init will try to install it which will not work in
          # isolated environments.
          - bridge-utils
-         - linux-firmware
-         - grub2-efi-modules
-         - efibootmgr
          # Tools needed to allow custom storage to be deployed without
          # accessing the Internet.
+         # grub2-efi-x64 contains the signed bootloader which currently does not
+         # chainboot - RHBZ #1623296
+         # - grub2-efi-x64
+         # - shim-x64
+         # RHBZ #1101352
+         - grub2-efi-x64-modules
+         - grub2-tools
+         - efibootmgr
          - dosfstools
          - lvm2
          - mdadm
@@ -57,6 +63,7 @@
          - bridge-utils
          # Tools needed to allow custom storage to be deployed without
          # accessing the Internet.
+         - efibootmgr
          - btrfs-progs
          - dosfstools
          - lvm2

=== modified file 'curtin/centos7/curtin-hooks.py'
--- curtin/centos7/curtin-hooks.py	2017-08-02 19:10:25 +0000
+++ curtin/centos7/curtin-hooks.py	2018-08-29 01:11:32 +0000
@@ -7,6 +7,7 @@
     )
 
 import codecs
+import importlib
 import os
 import re
 import sys
@@ -14,41 +15,37 @@
 from curtin import (
     block,
     config,
+    log,
     util,
     )
 
-try:
-    from curtin import FEATURES as curtin_features
-except ImportError:
-    curtin_features = []
-
-write_files = None
-try:
-    from curtin.futil import write_files
-except ImportError:
-    pass
-
-centos_apply_network_config = None
-try:
-    if 'CENTOS_APPLY_NETWORK_CONFIG' in curtin_features:
-        from curtin.commands.curthooks import centos_apply_network_config
-except ImportError:
-    pass
-
-"""
-CentOS 7
-
-Currently Support:
-
-- Legacy boot
-- UEFI boot
-- DHCP of BOOTIF
-
-Not Supported:
-
-- Multiple network configration
-- IPv6
-"""
+
+def try_import(path, unavailable=None, condition=True):
+    """Import path (module.submodule.attribute) and return it.
+    return unavailable if
+      - condition is not true
+      - import fails
+      - attribute does not exist in module."""
+    if not condition:
+        return unavailable
+    modname, _, attrname = path.rpartition(".")
+    try:
+        mod = importlib.import_module(modname)
+        imported = getattr(mod, attrname, None)
+    except ImportError:
+        imported = None
+    return imported if imported else unavailable
+
+
+curtin_features = try_import("curtin.FEATURES", [])
+write_files = try_import("curtin.futil.write_files")
+centos_apply_network_config = try_import(
+    "curtin.commands.curthooks.centos_apply_network_config",
+    condition='CENTOS_APPLY_NETWORK_CONFIG' in curtin_features)
+centos_curthooks = try_import(
+    "curtin.commands.curthooks.builtin_curthooks",
+    condition='CENTOS_CURTHOOK_SUPPORT' in curtin_features)
+
 
 FSTAB_PREPEND = """\
 #
@@ -342,6 +339,10 @@
 
 
 def main():
+    # Enable logging to stdout for curtin functions
+    verbosity = int(os.environ.get("CURTIN_VERBOSITY", 1))
+    log.basicConfig(stream=sys.stdout, verbosity=verbosity)
+
     state = util.load_command_environment()
     target = state['target']
     if target is None:
@@ -360,6 +361,19 @@
         print("Unable to find block device for: %s" % target)
         sys.exit(1)
 
+    if state.get('config'):
+        cfg = config.load_config(state['config'])
+    else:
+        cfg = {}
+
+    # Run builtin hooks if curtin has the feature and maas sent storage cfg
+    if centos_curthooks and cfg.get('storage'):
+        # centos_curthooks will complete successfully or raise an exception.
+        # Not handling the exception here means we will exit this program with
+        # a non-zero value
+        centos_curthooks(cfg, target, state)
+        return
+
     write_fstab(target, fstab)
 
     update_grub_default(
@@ -373,11 +387,6 @@
 
     set_autorelabel(target)
 
-    if state.get('config'):
-        cfg = config.load_config(state['config'])
-    else:
-        cfg = {}
-
     handle_cloudconfig(cfg, target)
 
     apply_networking(cfg, target, bootmac)