Merge ~raharper/curtin:fix/zfsroot-export-zpools-after-install into curtin:master

[Ryan Harper]

install: export zpools if present in the storage-config

Curtin needs to run zpool export on the any pool it created
before booting as zfs-initramfs 0.8 now requires a pool to
be exported before being imported.

LP: #1838278

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:e4d182392a51c636e348fafb7ae621ae765f5c36
https://jenkins.ubuntu.com/server/job/curtin-ci/3698/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3698/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3698/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3698/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3698/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3698//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:86405219079070391ee446f9d6ce7d0d24fad2c2
https://jenkins.ubuntu.com/server/job/curtin-ci/3699/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3699/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3699/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3699/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3699/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3699//rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

You should mention in commit message that you're putting /run into all chroots now.

[Ryan Harper (raharper)]

> You should mention in commit message that you're putting /run into all chroots
> now.

Fixed now; I didn't mean to include that in this merge. I'll put the /run one up separately.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:21b15a33527b2d515d07724cd055b1b7d3f6e539
https://jenkins.ubuntu.com/server/job/curtin-ci/3700/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3700/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3700/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3700/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3700/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3700//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:21b15a33527b2d515d07724cd055b1b7d3f6e539
https://jenkins.ubuntu.com/server/job/curtin-ci/3701/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3701/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3701/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3701/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3701/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3701//rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

vmtest shows that the fstype: zfsroot scenario needs a fix too.

[Ryan Harper (raharper)]

Thanks, I'll fix up the early return. And I need to add a fix for the zfsroot fstype case.

[Ryan Harper (raharper)]

Computer says "Yes!"

https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-debug/191/console

----------------------------------------------------------------------
Ran 182 tests in 1687.305s

OK (SKIP=28)
Wed, 18 Sep 2019 13:15:40 +0000: vmtest end [0] in 1690s

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:9005042fc4db0d31876ed361354e554f46d3fe30
https://jenkins.ubuntu.com/server/job/curtin-ci/3703/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3703/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3703/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3703/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3703/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3703//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:9005042fc4db0d31876ed361354e554f46d3fe30
https://jenkins.ubuntu.com/server/job/curtin-ci/3704/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3704/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3704/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3704/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3704/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3704//rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

The /run change got moved https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/372942

Thanks for separating.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:8c362d2e8dcfb26792c7978f6528c90e9845d48a
https://jenkins.ubuntu.com/server/job/curtin-ci/3717/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3717/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3717/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3717/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3717/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3717//rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Thanks for this, looks good Ryan, just one question about limiting where we need to do this to Eoan ++,

review: Needs Information

[Chad Smith (chad.smith)]

+1 on this. Thanks for the response.

review: Approve

diff --git a/curtin/block/zfs.py b/curtin/block/zfs.py
index dce15c3..25751d3 100644
--- a/curtin/block/zfs.py
+++ b/curtin/block/zfs.py
@@ -281,4 +281,28 @@ def device_to_poolname(devname):
     if vdev_type == 'zfs_member' and label:
         return label
 
+
+def get_zpool_from_config(cfg):
+    """Parse a curtin storage config and return a list
+       of zpools that were created.
+    """
+    if not cfg:
+        return []
+
+    if 'storage' not in cfg:
+        return []
+
+    zpools = []
+    sconfig = cfg['storage']['config']
+    for item in sconfig:
+        if item['type'] == 'zpool':
+            zpools.append(item['pool'])
+        elif item['type'] == 'format':
+            if item['fstype'] == 'zfsroot':
+                # curtin.commands.blockmeta sets pool='rpool' for zfsroot
+                zpools.append('rpool')
+
+    return zpools
+
+
 # vi: ts=4 expandtab syntax=python
diff --git a/curtin/commands/install.py b/curtin/commands/install.py
index ad17508..a3471f6 100644
--- a/curtin/commands/install.py
+++ b/curtin/commands/install.py
@@ -11,7 +11,7 @@ import subprocess
 import sys
 import tempfile
 
-from curtin.block import iscsi
+from curtin.block import iscsi, zfs
 from curtin import config
 from curtin import distro
 from curtin import util
@@ -505,6 +505,10 @@ def cmd_install(args):
             if iscsi.get_iscsi_disks_from_config(cfg):
                 iscsi.restart_iscsi_service()
 
+            for pool in zfs.get_zpool_from_config(cfg):
+                LOG.debug('Exporting ZFS zpool %s', pool)
+                zfs.zpool_export(pool)
+
             shutil.rmtree(workingd.top)
 
     apply_power_state(cfg.get('power_state'))
diff --git a/tests/unittests/test_block_zfs.py b/tests/unittests/test_block_zfs.py
index f45fde2..3508d4b 100644
--- a/tests/unittests/test_block_zfs.py
+++ b/tests/unittests/test_block_zfs.py
@@ -538,4 +538,30 @@ class TestZfsSupported(CiTestCase):
         self.assertEqual(1, m_assert_zfs.call_count)
 
 
+class TestZfsGetPoolFromConfig(CiTestCase):
+
+    def test_get_zpool_from_config_returns_empty(self):
+        """ get_zpool_from_config_returns_empty for invalid/empty configs."""
+        configs = (None, {}, {'network': 'foobar'},
+                   {'storage': {'config': []}})
+        for cfg in configs:
+            self.assertEqual([],
+                             zfs.get_zpool_from_config(cfg))
+
+    def test_get_zpool_from_config(self):
+        """ get_zpool_from_config_returns pool names for each zpool in cfg."""
+        expected_zpools = ['rpool1', 'rpool2']
+        zpool_cfg = [{'type': 'zpool', 'pool': pool}
+                     for pool in expected_zpools]
+        sconfig = {'storage': {'config': zpool_cfg}}
+        self.assertEqual(sorted(expected_zpools),
+                         sorted(zfs.get_zpool_from_config(sconfig)))
+
+    def test_get_zpool_from_config_zfsroot(self):
+        """ get_zpool_from_config_returns injected pool name for zfsroot."""
+        zpool_cfg = [{'type': 'format', 'fstype': 'zfsroot'}]
+        sconfig = {'storage': {'config': zpool_cfg}}
+        self.assertEqual(['rpool'], zfs.get_zpool_from_config(sconfig))
+
+
 # vi: ts=4 expandtab syntax=python
diff --git a/tests/vmtests/test_zfsroot.py b/tests/vmtests/test_zfsroot.py
index a2fc316..a0e50e9 100644
--- a/tests/vmtests/test_zfsroot.py
+++ b/tests/vmtests/test_zfsroot.py
@@ -99,7 +99,6 @@ class DiscoTestZfsRoot(relbase.disco, TestZfsRootAbs):
     mem = 4096
 
 
[e-mail]_by_date("1838278", fixby="2019-10-01", install=False)
 class EoanTestZfsRoot(relbase.eoan, TestZfsRootAbs):
     __test__ = True
     mem = 4096
@@ -129,7 +128,6 @@ class DiscoTestZfsRootFsType(relbase.disco, TestZfsRootFsTypeAbs):
     mem = 4096
 
 
[e-mail]_by_date("1838278", fixby="2019-10-01", install=False)
 class EoanTestZfsRootFsType(relbase.eoan, TestZfsRootFsTypeAbs):
     __test__ = True
     mem = 4096