Merge ~raharper/curtin:fix/support-i386 into curtin:master

[Ryan Harper]

zfs: implement a supported check to handle i386

The zfs kernel module is not available for i386 so curtin cannot
expect that missing that module is fatal. Concentrate the various
checks for when we load the zfs kernel module into the zfs python
module. Add vmtest to exercise the i386 path.

LP: #1768709

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:697b47a6b5db12969527992548ee2515309da5ed
https://jenkins.ubuntu.com/server/job/curtin-ci/920/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/920/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/920/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/920/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/920/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/920/rebuild

review: Needs Fixing (continuous-integration)

99e6520... by Ryan Harper on 2018-05-09

Fix tox

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:4e2dfdcf9868d98702ca3fc785727d45039b69a6
https://jenkins.ubuntu.com/server/job/curtin-ci/921/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/921
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/921
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/921
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/921

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/921/rebuild

review: Approve (continuous-integration)

21715ca... by Ryan Harper on 2018-05-09

Fix skip_if_expected_failure implementation

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:a18196af13d9718f293cd5fd2c1df3f620b52286
https://jenkins.ubuntu.com/server/job/curtin-ci/922/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/922
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/922
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/922
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/922

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/922/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

So essentially, you have to decorate all tests that would be run
by an expected failure run with the 'skip_if_flag'.

it seems heavy to decorate in a superclass of all tests
for this small usecase.

I like the other changes, and I'm not strictly opposed to this one, but it just seems heavy handed.

[Ryan Harper (raharper)]

I think doing it in the base class protects against when we add new base-class tests that would fail, we'd need to add a skip in the zfsroot class which could have copied all of the base class tests and decorated them.

I'm open to alternatives that are lighter if you have them; and we may want to create new expected_failure classes to test error paths, in which case we'd want something like this anyhow.

9c3eab6... by Ryan Harper on 2018-05-09

Fix missing space in RuntimeError exception.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c566934eeebcf1db982d7f9bdf857cbb0745eae8
https://jenkins.ubuntu.com/server/job/curtin-ci/923/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/923
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/923
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/923
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/923

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/923/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I wont block you on this, but subsequently I'll look at a lighter maintenance-wise version of expectedFailure.

[Scott Moser (smoser)]

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c05b4828745c18dd5381f114adf10f7548a072d4
https://jenkins.ubuntu.com/server/job/curtin-ci/929/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/929
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/929
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/929
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/929

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/929/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Started vmtest run on jenkins here (full run since we touch BaseVMClass ).

https://jenkins.ubuntu.com/server/view/curtin/job/curtin-vmtest-devel-debug/61/console

[Ryan Harper (raharper)]

Bah, the i386 vmtests failed since we don't have the i386 images synced yet. So instead I'll do a full run on diglett and post results here.

[Ryan Harper (raharper)]

Looks like I need to add some code to detect the arch values in the test-cases much like we do for distro/release so that 'make sync-images' pulls in the i386 variants.

Fixing that now.

c55b507... by Ryan Harper on 2018-05-15

vmtest-sync-images: add find_arches() to sync all test images

Introduction of i386 tests means that when generating the mirroring
filter vmtest needs to find all values of the 'arch' attribute.
Refactor generating the list of testcases into common find_testcases()

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:9ddc08422f21e39dbaeddbf58a6cf097bf3e1c86
https://jenkins.ubuntu.com/server/job/curtin-ci/930/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/930
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/930
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/930
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/930

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/930/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Diglett vmtest run of this branch is successful after sync'ing i386 images.

----------------------------------------------------------------------
Ran 3098 tests in 15824.715s

OK (SKIP=249)
Tue, 15 May 2018 18:46:25 -0500: vmtest end [0] in 15826s

92f2032... by Ryan Harper on 2018-05-17

Add unittests for zfs.zfs_supported and update mock for clear-holders use of zfs_supported

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:382556653e5ef8ae18ea83ce899b71e859fd9ff7
https://jenkins.ubuntu.com/server/job/curtin-ci/934/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/934
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/934
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/934
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/934

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/934/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Somehow the unittests had started to fail when I went to rebase this on master. I rebased and then applied a refactor to the unittests. I'm re-running vmtest on diglett.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:92f2032d3a6fed961a51aaa7ff7f7eb3e7c65fea
https://jenkins.ubuntu.com/server/job/curtin-ci/937/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/937
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/937
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/937
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/937

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/937/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Clean vmtest run after rebase and unittest fixes:

% rm -rf output/; CURTIN_VMTEST_PARALLEL=4 CURTIN_VMTEST_BOOT_TIMEOUT=300 CURTIN_VMTEST_KEEP_DATA_FAIL=all ./tools/jenkins-runner -vv --nologcapture tests/vmtests/test_zfsroot.py
CURTIN_VMTEST_BOOT_TIMEOUT=300
CURTIN_VMTEST_IMAGE_SYNC=0
CURTIN_VMTEST_ISCSI_PORTAL=10.245.168.20:1202
CURTIN_VMTEST_KEEP_DATA_FAIL=all
CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect
CURTIN_VMTEST_LOG=/home/rharper/work/git/curtin/output/debug.log
CURTIN_VMTEST_PARALLEL=4
CURTIN_VMTEST_REUSE_TOPDIR=0
CURTIN_VMTEST_TAR_DISKS=0
CURTIN_VMTEST_TOPDIR=/home/rharper/work/git/curtin/output
TGT_IPC_SOCKET=/home/rharper/work/git/curtin/output/tgt.d/socket
TGT_LOG_D=/home/rharper/work/git/curtin/output/tgt.d
TGT_PID=14721
TGT_PORTAL=10.245.168.20:1202
http_proxy=http://squid.internal:3128/
https_proxy=http://squid.internal:3128/
no_proxy=
Quering synced ephemeral images/kernels in /srv/images
======================================================================================
 Release Codename ImageDate Arch/SubArch Path
--------------------------------------------------------------------------------------
   12.04 precise 20170424 amd64/hwe-t precise/amd64/20170424/root-image.gz
   12.04 precise 20170424.1 amd64/hwe-p precise/amd64/20170424/squashfs
   14.04 trusty 20180502 amd64/hwe-t trusty/amd64/20180502/squashfs
   14.04 trusty 20180502 amd64/hwe-x trusty/amd64/20180502/squashfs
   14.04 trusty 20180502 i386/hwe-t trusty/i386/20180502/squashfs
   14.04 trusty 20180502 i386/hwe-x trusty/i386/20180502/squashfs
   16.04 xenial 20180511 amd64/ga-16.04 xenial/amd64/20180511/squashfs
   16.04 xenial 20180511 amd64/hwe-16.04 xenial/amd64/20180511/squashfs
   16.04 xenial 20180511 amd64/hwe-16.04-edge xenial/amd64/20180511/squashfs
   16.04 xenial 20180511 i386/ga-16.04 xenial/i386/20180511/squashfs
   16.04 xenial 20180511 i386/hwe-16.04 xenial/i386/20180511/squashfs
   16.04 xenial 20180511 i386/hwe-16.04-edge xenial/i386/20180511/squashfs
   17.04 zesty 20171219 amd64/ga-17.04 zesty/amd64/20171219/squashfs
   17.10 artful 20180516 amd64/ga-17.10 artful/amd64/20180516/squashfs
   18.04 bionic 20180509 amd64/ga-18.04 bionic/amd64/20180509/squashfs
   18.04 bionic 20180509 i386/ga-18.04 bionic/i386/20180509/squashfs
======================================================================================

...

----------------------------------------------------------------------
Ran 3109 tests in 15897.269s

OK (SKIP=249)
Thu, 17 May 2018 18:07:47 -0500: vmtest end [0] in 15899s

[Ryan Harper (raharper)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=5313b9e2

diff --git a/curtin/block/clear_holders.py b/curtin/block/clear_holders.py
index cbb7074..20c572b 100644
--- a/curtin/block/clear_holders.py
+++ b/curtin/block/clear_holders.py
@@ -610,8 +610,11 @@ def start_clear_holders_deps():
     util.load_kernel_module('bcache')
     # the zfs module is needed to find and export devices which may be in-use
     # and need to be cleared, only on xenial+.
-    if not util.lsb_release()['codename'] in ['precise', 'trusty']:
-        util.load_kernel_module('zfs')
+    try:
+        if zfs.zfs_supported():
+            util.load_kernel_module('zfs')
+    except RuntimeError as e:
+        LOG.warning('Failed to load zfs kernel module: %s', e)
 
 
 # anything that is not identified can assumed to be a 'disk' or similar
diff --git a/curtin/block/zfs.py b/curtin/block/zfs.py
index 8e00c7e..cfb07a9 100644
--- a/curtin/block/zfs.py
+++ b/curtin/block/zfs.py
@@ -21,6 +21,9 @@ ZFS_DEFAULT_PROPERTIES = {
     'normalization': 'formD',
 }
 
+ZFS_UNSUPPORTED_ARCHES = ['i386']
+ZFS_UNSUPPORTED_RELEASES = ['precise', 'trusty']
+
 
 def _join_flags(optflag, params):
     """
@@ -69,6 +72,28 @@ def _join_pool_volume(poolname, volume):
     return os.path.normpath("%s/%s" % (poolname, volume))
 
 
+def zfs_supported():
+    """ Determine if the runtime system supports zfs.
+    returns: True if system supports zfs
+    raises: RuntimeError: if system does not support zfs
+    """
+    arch = util.get_platform_arch()
+    if arch in ZFS_UNSUPPORTED_ARCHES:
+        raise RuntimeError("zfs is not supported on architecture: %s" % arch)
+
+    release = util.lsb_release()['codename']
+    if release in ZFS_UNSUPPORTED_RELEASES:
+        raise RuntimeError("zfs is not supported on release: %s" % release)
+
+    try:
+        util.subp(['modinfo', 'zfs'], capture=True)
+    except util.ProcessExecutionError as err:
+        if err.stderr.startswith("modinfo: ERROR: Module zfs not found."):
+            raise RuntimeError("zfs kernel module is not available: %s" % err)
+
+    return True
+
+
 def zpool_create(poolname, vdevs, mountpoint=None, altroot=None,
                  pool_properties=None, zfs_properties=None):
     """
diff --git a/curtin/commands/block_meta.py b/curtin/commands/block_meta.py
index ff2ab80..f5b82cf 100644
--- a/curtin/commands/block_meta.py
+++ b/curtin/commands/block_meta.py
@@ -1263,6 +1263,8 @@ def zpool_handler(info, storage_config):
     """
     Create a zpool based in storage_configuration
     """
+    zfs.zfs_supported()
+
     state = util.load_command_environment()
 
     # extract /dev/disk/by-id paths for each volume used
@@ -1296,6 +1298,7 @@ def zfs_handler(info, storage_config):
     """
     Create a zfs filesystem
     """
+    zfs.zfs_supported()
     state = util.load_command_environment()
     poolname = get_poolname(info, storage_config)
     volume = info.get('volume')
diff --git a/tests/unittests/test_block_zfs.py b/tests/unittests/test_block_zfs.py
index 883f727..c61a6da 100644
--- a/tests/unittests/test_block_zfs.py
+++ b/tests/unittests/test_block_zfs.py
@@ -1,5 +1,8 @@
+import mock
+
 from curtin.config import merge_config
 from curtin.block import zfs
+from curtin.util import ProcessExecutionError
 from .helpers import CiTestCase
 
 
@@ -375,4 +378,97 @@ class TestBlockZfsDeviceToPoolname(CiTestCase):
         self.mock_blkid.assert_called_with(devs=[devname])
 
 
+class TestBlockZfsZfsSupported(CiTestCase):
+
+    def setUp(self):
+        super(TestBlockZfsZfsSupported, self).setUp()
+        self.add_patch('curtin.block.zfs.util.subp', 'mock_subp')
+        self.add_patch('curtin.block.zfs.util.get_platform_arch', 'mock_arch')
+        self.add_patch('curtin.block.zfs.util.lsb_release', 'mock_release')
+        self.mock_release.return_value = {'codename': 'xenial'}
+        self.mock_arch.return_value = 'x86_64'
+
+    def test_supported_arch(self):
+        self.assertTrue(zfs.zfs_supported())
+
+    def test_unsupported_arch(self):
+        self.mock_arch.return_value = 'i386'
+        with self.assertRaises(RuntimeError):
+            zfs.zfs_supported()
+
+    def test_unsupported_releases(self):
+        for rel in ['precise', 'trusty']:
+            self.mock_release.return_value = {'codename': rel}
+            with self.assertRaises(RuntimeError):
+                zfs.zfs_supported()
+
+    def test_missing_module(self):
+        missing = 'modinfo: ERROR: Module zfs not found.\n     '
+        self.mock_subp.side_effect = ProcessExecutionError(stdout='',
+                                                           stderr=missing,
+                                                           exit_code='1')
+        with self.assertRaises(RuntimeError):
+            zfs.zfs_supported()
+
+
+class TestZfsSupported(CiTestCase):
+
+    def setUp(self):
+        super(TestZfsSupported, self).setUp()
+
+    @mock.patch('curtin.block.zfs.util')
+    def test_zfs_supported_returns_true(self, mock_util):
+        """zfs_supported returns True on supported platforms"""
+        mock_util.get_platform_arch.return_value = 'amd64'
+        mock_util.lsb_release.return_value = {'codename': 'bionic'}
+        mock_util.subp.return_value = ("", "")
+
+        self.assertNotIn(mock_util.get_platform_arch.return_value,
+                         zfs.ZFS_UNSUPPORTED_ARCHES)
+        self.assertNotIn(mock_util.lsb_release.return_value['codename'],
+                         zfs.ZFS_UNSUPPORTED_RELEASES)
+        self.assertTrue(zfs.zfs_supported())
+
+    @mock.patch('curtin.block.zfs.util')
+    def test_zfs_supported_raises_exception_on_bad_arch(self, mock_util):
+        """zfs_supported raises RuntimeError on unspported arches"""
+        mock_util.lsb_release.return_value = {'codename': 'bionic'}
+        mock_util.subp.return_value = ("", "")
+        for arch in zfs.ZFS_UNSUPPORTED_ARCHES:
+            mock_util.get_platform_arch.return_value = arch
+            with self.assertRaises(RuntimeError):
+                zfs.zfs_supported()
+
+    @mock.patch('curtin.block.zfs.util')
+    def test_zfs_supported_raises_execption_on_bad_releases(self, mock_util):
+        """zfs_supported raises RuntimeError on unspported releases"""
+        mock_util.get_platform_arch.return_value = 'amd64'
+        mock_util.subp.return_value = ("", "")
+        for release in zfs.ZFS_UNSUPPORTED_RELEASES:
+            mock_util.lsb_release.return_value = {'codename': release}
+            with self.assertRaises(RuntimeError):
+                zfs.zfs_supported()
+
+    @mock.patch('curtin.block.zfs.util.subprocess.Popen')
+    @mock.patch('curtin.block.zfs.util.lsb_release')
+    @mock.patch('curtin.block.zfs.util.get_platform_arch')
+    def test_zfs_supported_raises_exception_on_missing_module(self,
+                                                              m_arch,
+                                                              m_release,
+                                                              m_popen):
+        """zfs_supported raises RuntimeError on missing zfs module"""
+
+        m_arch.return_value = 'amd64'
+        m_release.return_value = {'codename': 'bionic'}
+        process_mock = mock.Mock()
+        attrs = {
+            'returncode': 1,
+            'communicate.return_value':
+                ('output', "modinfo: ERROR: Module zfs not found."),
+        }
+        process_mock.configure_mock(**attrs)
+        m_popen.return_value = process_mock
+        with self.assertRaises(RuntimeError):
+            zfs.zfs_supported()
+
 # vi: ts=4 expandtab syntax=python
diff --git a/tests/unittests/test_clear_holders.py b/tests/unittests/test_clear_holders.py
index 9c93b5a..ceb5615 100644
--- a/tests/unittests/test_clear_holders.py
+++ b/tests/unittests/test_clear_holders.py
@@ -723,29 +723,31 @@ class TestClearHolders(CiTestCase):
         mock_gen_holders_tree.return_value = self.example_holders_trees[1][1]
         clear_holders.assert_clear(device)
 
+    @mock.patch('curtin.block.clear_holders.zfs')
     @mock.patch('curtin.block.clear_holders.mdadm')
     @mock.patch('curtin.block.clear_holders.util')
-    def test_start_clear_holders_deps(self, mock_util, mock_mdadm):
-        mock_util.lsb_release.return_value = {'codename': 'xenial'}
+    def test_start_clear_holders_deps(self, mock_util, mock_mdadm, mock_zfs):
+        mock_zfs.zfs_supported.return_value = True
         clear_holders.start_clear_holders_deps()
         mock_mdadm.mdadm_assemble.assert_called_with(
             scan=True, ignore_errors=True)
         mock_util.load_kernel_module.assert_has_calls([
                 mock.call('bcache'), mock.call('zfs')])
 
+    @mock.patch('curtin.block.clear_holders.zfs')
     @mock.patch('curtin.block.clear_holders.mdadm')
     @mock.patch('curtin.block.clear_holders.util')
-    def test_start_clear_holders_deps_nozfs(self, mock_util, mock_mdadm):
-        """ test that we skip zfs modprobe on precise, trusty """
-        for codename in ['precise', 'trusty']:
-            mock_util.lsb_release.return_value = {'codename': codename}
-            clear_holders.start_clear_holders_deps()
-            mock_mdadm.mdadm_assemble.assert_called_with(
-                scan=True, ignore_errors=True)
-            mock_util.load_kernel_module.assert_has_calls(
-                [mock.call('bcache')])
-            self.assertNotIn(mock.call('zfs'),
-                             mock_util.load_kernel_module.call_args_list)
+    def test_start_clear_holders_deps_nozfs(self, mock_util, mock_mdadm,
+                                            mock_zfs):
+        """test that we skip zfs modprobe on unsupported platforms"""
+        mock_zfs.zfs_supported.return_value = False
+        clear_holders.start_clear_holders_deps()
+        mock_mdadm.mdadm_assemble.assert_called_with(
+            scan=True, ignore_errors=True)
+        mock_util.load_kernel_module.assert_has_calls(
+            [mock.call('bcache')])
+        self.assertNotIn(mock.call('zfs'),
+                         mock_util.load_kernel_module.call_args_list)
 
     @mock.patch('curtin.block.clear_holders.util')
     def test_shutdown_swap_calls_swapoff(self, mock_util):
diff --git a/tests/vmtests/__init__.py b/tests/vmtests/__init__.py
index d02ad33..5c30a83 100644
--- a/tests/vmtests/__init__.py
+++ b/tests/vmtests/__init__.py
@@ -350,8 +350,23 @@ class TempDir(object):
                               stdout=DEVNULL, stderr=subprocess.STDOUT)
 
 
+def skip_if_flag(flag):
+    def decorator(func):
+        """the name test_wrapper below has to start with test, or nose's
+           filter will not run it."""
+        def test_wrapper(self, *args, **kwargs):
+            val = getattr(self, flag, None)
+            if val:
+                self.skipTest("skip due to %s=%s" % (flag, val))
+            else:
+                return func(self, *args, **kwargs)
+        return test_wrapper
+    return decorator
+
+
 class VMBaseClass(TestCase):
     __test__ = False
+    expected_failure = False
     arch_skip = []
     boot_timeout = BOOT_TIMEOUT
     collect_scripts = [textwrap.dedent("""
@@ -959,6 +974,10 @@ class VMBaseClass(TestCase):
             else:
                 logger.warn("Boot for install did not produce a console log.")
 
+        if cls.expected_failure:
+            logger.debug('Expected Failure: skipping boot stage')
+            return
+
         logger.debug('')
         try:
             if os.path.exists(cls.install_log):
@@ -1302,6 +1321,7 @@ class VMBaseClass(TestCase):
             ret[val[0]] = val[1]
         return ret
 
+    @skip_if_flag('expected_failure')
     def test_fstab(self):
         if self.fstab_expected is None:
             return
@@ -1317,6 +1337,7 @@ class VMBaseClass(TestCase):
                         self.assertEqual(fstab_entry.split(' ')[1],
                                          mntpoint)
 
+    @skip_if_flag('expected_failure')
     def test_dname(self, disk_to_check=None):
         if "trusty" in [self.release, self.target_release]:
             raise SkipTest(
@@ -1339,6 +1360,7 @@ class VMBaseClass(TestCase):
                 self.assertIn(link, contents)
             self.assertIn(diskname, contents)
 
+    @skip_if_flag('expected_failure')
     def test_reporting_data(self):
         with open(self.reporting_log, 'r') as fp:
             data = json.load(fp)
@@ -1358,6 +1380,7 @@ class VMBaseClass(TestCase):
         self.assertIn('path', files)
         self.assertEqual('/tmp/install.log', files.get('path', ''))
 
+    @skip_if_flag('expected_failure')
     def test_interfacesd_eth0_removed(self):
         """ Check that curtin has removed /etc/network/interfaces.d/eth0.cfg
             by examining the output of a find /etc/network > find_interfaces.d
@@ -1366,9 +1389,9 @@ class VMBaseClass(TestCase):
         self.assertNotIn("/etc/network/interfaces.d/eth0.cfg",
                          interfacesd.split("\n"))
 
+    @skip_if_flag('expected_failure')
     def test_installed_correct_kernel_package(self):
         """ Test curtin installs the correct kernel package.  """
-
         # target_distro is set for non-ubuntu targets
         if self.target_distro is not None:
             raise SkipTest("Can't check non-ubuntu kernel packages")
@@ -1415,6 +1438,7 @@ class VMBaseClass(TestCase):
             self._debian_packages = pkgs
         return self._debian_packages
 
+    @skip_if_flag('expected_failure')
     def test_swaps_used(self):
         cfg = yaml.load(self.load_collect_file("root/curtin-install-cfg.yaml"))
         stgcfg = cfg.get("storage", {}).get("config", [])
@@ -1553,14 +1577,19 @@ def get_rfc4173(ip, port, target, user=None, pword=None,
 
 
 def find_error_context(err_match, contents, nrchars=200):
+    traceback_end = re.compile(r'Error:.*')
+    end_match = traceback_end.search(contents, err_match.start())
     context_start = err_match.start() - nrchars
-    context_end = err_match.end() + nrchars
+    if end_match:
+        context_end = end_match.end()
+    else:
+        context_end = err_match.end() + nrchars
     # extract contents, split into lines, drop the first and last partials
     # recombine and return
     return "\n".join(contents[context_start:context_end].splitlines()[1:-1])
 
 
-def check_install_log(install_log):
+def check_install_log(install_log, nrchars=200):
     # look if install is OK via curtin 'Installation ok"
     # if we dont find that, scan for known error messages and report
     # if we don't see any errors, fail with general error
@@ -1574,7 +1603,7 @@ def check_install_log(install_log):
                    'ImportError: No module named.*',
                    'Unexpected error while running command',
                    'E: Unable to locate package.*',
-                   'Traceback.*most recent call last.*:']))
+                   'cloud-init.*: Traceback.*']))
 
     install_is_ok = re.findall(install_pass, install_log)
     # always scan for errors
@@ -1583,7 +1612,7 @@ def check_install_log(install_log):
         errmsg = ('Failed to verify Installation is OK')
 
     for e in found_errors:
-        errors.append(find_error_context(e, install_log))
+        errors.append(find_error_context(e, install_log, nrchars=nrchars))
         errmsg = ('Errors during curtin installer')
 
     return errmsg, errors
diff --git a/tests/vmtests/helpers.py b/tests/vmtests/helpers.py
index 7fc92e1..10e20b3 100644
--- a/tests/vmtests/helpers.py
+++ b/tests/vmtests/helpers.py
@@ -86,18 +86,7 @@ def check_call(cmd, signal=signal.SIGTERM, **kwargs):
     return Command(cmd, signal).run(**kwargs)
 
 
-def find_releases_by_distro():
-    """
-    Returns a dictionary of distros and the distro releases that will be tested
-
-    distros:
-        ubuntu:
-            releases: []
-            krels: []
-        centos:
-            releases: []
-            krels: []
-    """
+def find_testcases():
     # Use the TestLoder to load all test cases defined within tests/vmtests/
     # and figure out what distros and releases they are testing. Any tests
     # which are disabled will be excluded.
@@ -108,32 +97,60 @@ def find_releases_by_distro():
     root_dir = os.path.split(os.path.split(tests_dir)[0])[0]
     # Find all test modules defined in curtin/tests/vmtests/
     module_test_suites = loader.discover(tests_dir, top_level_dir=root_dir)
-    # find all distros and releases tested for each distro
-    releases = []
-    krels = []
-    rel_by_dist = {}
     for mts in module_test_suites:
         for class_test_suite in mts:
             for test_case in class_test_suite:
                 # skip disabled tests
                 if not getattr(test_case, '__test__', False):
                     continue
-                for (dist, rel, krel) in (
-                        (getattr(test_case, a, None) for a in attrs)
-                        for attrs in (('distro', 'release', 'krel'),
-                                      ('target_distro', 'target_release',
-                                       'krel'))):
-
-                    if dist and rel:
-                        distro = rel_by_dist.get(dist, {'releases': [],
-                                                        'krels': []})
-                        releases = distro.get('releases')
-                        krels = distro.get('krels')
-                        if rel not in releases:
-                            releases.append(rel)
-                        if krel and krel not in krels:
-                            krels.append(krel)
-                        rel_by_dist.update({dist: distro})
+                yield test_case
+
+
+def find_arches():
+    """
+    Return a list of uniq arch values from test cases
+    """
+    arches = []
+    for test_case in find_testcases():
+        arch = getattr(test_case, 'arch', None)
+        if arch and arch not in arches:
+            arches.append(arch)
+    return arches
+
+
+def find_releases_by_distro():
+    """
+    Returns a dictionary of distros and the distro releases that will be tested
+
+    distros:
+        ubuntu:
+            releases: []
+            krels: []
+        centos:
+            releases: []
+            krels: []
+    """
+    # find all distros and releases tested for each distro
+    releases = []
+    krels = []
+    rel_by_dist = {}
+    for test_case in find_testcases():
+        for (dist, rel, krel) in (
+                (getattr(test_case, a, None) for a in attrs)
+                for attrs in (('distro', 'release', 'krel'),
+                              ('target_distro', 'target_release',
+                               'krel'))):
+
+            if dist and rel:
+                distro = rel_by_dist.get(dist, {'releases': [],
+                                                'krels': []})
+                releases = distro.get('releases')
+                krels = distro.get('krels')
+                if rel not in releases:
+                    releases.append(rel)
+                if krel and krel not in krels:
+                    krels.append(krel)
+                rel_by_dist.update({dist: distro})
 
     return rel_by_dist
 
diff --git a/tests/vmtests/test_basic.py b/tests/vmtests/test_basic.py
index ce86406..2e47cb6 100644
--- a/tests/vmtests/test_basic.py
+++ b/tests/vmtests/test_basic.py
@@ -152,6 +152,11 @@ class TrustyHWEXTestBasic(relbase.trusty_hwe_x, TrustyTestBasic):
     __test__ = True
 
 
+class XenialGAi386TestBasic(relbase.xenial_ga, TestBasicAbs):
+    __test__ = True
+    arch = 'i386'
+
+
 class XenialGATestBasic(relbase.xenial_ga, TestBasicAbs):
     __test__ = True
 
diff --git a/tests/vmtests/test_zfsroot.py b/tests/vmtests/test_zfsroot.py
index 84439d3..1ebc616 100644
--- a/tests/vmtests/test_zfsroot.py
+++ b/tests/vmtests/test_zfsroot.py
@@ -1,4 +1,4 @@
-from . import VMBaseClass
+from . import VMBaseClass, check_install_log, skip_if_flag
 from .releases import base_vm_classes as relbase
 
 import textwrap
@@ -33,6 +33,7 @@ class TestZfsRootAbs(VMBaseClass):
             echo "$v" > apt-proxy
         """)]
 
+    @skip_if_flag('expected_failure')
     def test_output_files_exist(self):
         self.output_files_exist(
             ["blkid_output_vda", "blkid_output_vda1", "blkid_output_vda2",
@@ -40,21 +41,49 @@ class TestZfsRootAbs(VMBaseClass):
              "proc_partitions",
              "root/curtin-install.log", "root/curtin-install-cfg.yaml"])
 
+    @skip_if_flag('expected_failure')
     def test_ptable(self):
         blkid_info = self.get_blkid_data("blkid_output_vda")
         self.assertEquals(blkid_info["PTTYPE"], "gpt")
 
+    @skip_if_flag('expected_failure')
     def test_zfs_list(self):
         """Check rpoot/ROOT/zfsroot is mounted at slash"""
         self.output_files_exist(['zfs_list'])
         self.check_file_regex('zfs_list', r"rpool/ROOT/zfsroot.*/\n")
 
+    @skip_if_flag('expected_failure')
     def test_proc_cmdline_has_root_zfs(self):
         """Check /proc/cmdline has root=ZFS=<pool>"""
         self.output_files_exist(['proc_cmdline'])
         self.check_file_regex('proc_cmdline', r"root=ZFS=rpool/ROOT/zfsroot")
 
 
+class UnsupportedZfs(VMBaseClass):
+    expected_failure = True
+    collect_scripts = []
+    interactive = False
+
+    def test_install_log_finds_zfs_runtime_error(self):
+        with open(self.install_log, 'rb') as lfh:
+            install_log = lfh.read().decode('utf-8', errors='replace')
+        errmsg, errors = check_install_log(install_log)
+        found_zfs = False
+        print("errors: %s" % (len(errors)))
+        for idx, err in enumerate(errors):
+            print("%s:\n%s" % (idx, err))
+            if 'RuntimeError' in err:
+                found_zfs = True
+                break
+        self.assertTrue(found_zfs)
+
+
+class XenialGAi386TestZfsRoot(relbase.xenial_ga, TestZfsRootAbs,
+                              UnsupportedZfs):
+    __test__ = True
+    arch = 'i386'
+
+
 class XenialGATestZfsRoot(relbase.xenial_ga, TestZfsRootAbs):
     __test__ = True
 
@@ -83,5 +112,11 @@ class XenialGATestZfsRootFsType(relbase.xenial_ga, TestZfsRootFsTypeAbs):
     __test__ = True
 
 
+class XenialGAi386TestZfsRootFsType(relbase.xenial_ga, TestZfsRootFsTypeAbs,
+                                    UnsupportedZfs):
+    __test__ = True
+    arch = 'i386'
+
+
 class BionicTestZfsRootFsType(relbase.bionic, TestZfsRootFsTypeAbs):
     __test__ = True
diff --git a/tools/vmtest-sync-images b/tools/vmtest-sync-images
index 26a1962..3d82b62 100755
--- a/tools/vmtest-sync-images
+++ b/tools/vmtest-sync-images
@@ -17,11 +17,9 @@ sys.path.insert(1, os.path.realpath(os.path.join(
 from tests.vmtests import (
     IMAGE_DIR, IMAGE_SRC_URL, sync_images)
 from tests.vmtests.image_sync import ITEM_NAME_FILTERS
-from tests.vmtests.helpers import find_releases_by_distro
+from tests.vmtests.helpers import (find_arches, find_releases_by_distro)
 from curtin.util import get_platform_arch
 
-DEFAULT_ARCH = get_platform_arch()
-
 
 def _fmt_list_filter(filter_name, matches):
     return '~'.join((filter_name, '|'.join(matches)))
@@ -53,7 +51,7 @@ if __name__ == '__main__':
                         os.unlink(fpath)
 
     arg_releases = [r for r in sys.argv[1:] if r != "--clean"]
-    arch_filters = ['arch={}'.format(DEFAULT_ARCH)]
+    arch_filters = [_fmt_list_filter('arch', find_arches())]
     filter_sets = []
     if len(arg_releases):
         filter_sets.append([_fmt_list_filter('release', arg_releases),