Merge lp:~raharper/curtin/trunk.snappy-dd into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

Add support for installing Ubuntu-Core 16 images

UC16 images are compressed-xz raw images which need to be written
directly to disk. Introduce new source types dd-{xz,gz,bz2,raw}
and pass them to block_meta write_image_to_disk. Update block_meta's
find_root_dev to look for UC16 specific paths to identify the root
partition.

UC16 images are generally not modifiable and do not contain dpkg/apt,
therefore curtin handles curthooks on UC16 differently. Add a new
top-level 'cloudconfig' key which is modeled after write_files to be used
by maas to inject cloud-init into a UC16 image; curtin appends these
files out in the "writable" partition of UC16 which will be mounted up on
top of the root partition. Add an initial vmtest (disabled by default) as
UC16 images are not yet available via maas streams data.

Add support for installing Ubuntu-Core 16 images

UC16 images are compressed-xz raw images which need to be written
directly to disk. Introduce new source types dd-{xz,gz,bz2,raw}
and pass them to block_meta write_image_to_disk. Update block_meta's
find_root_dev to look for UC16 specific paths to identify the root
partition.

UC16 images are generally not modifiable and do not contain dpkg/apt,
therefore curtin handles curthooks on UC16 differently. Add a new
top-level 'cloudconfig' key which is modeled after write_files to be used
by maas to inject cloud-init into a UC16 image; curtin appends these
files out in the "writable" partition of UC16 which will be mounted up on
top of the root partition. Add an initial vmtest; currently a hack since
UC16 images are not yet available via maas streams data.

[lp:~raharper/curtin/trunk.snappy-dd]

405. By Ryan Harper on 2017-02-17

Fix flake8 issues

[lp:~raharper/curtin/trunk.snappy-dd]

406. By Ryan Harper on 2017-03-21

refactor snappy_curthooks; add handle_cloudconfig hook

407. By Ryan Harper on 2017-03-21

merge from smoser snappy branch

408. By Ryan Harper on 2017-03-21

remove testing tools

409. By Ryan Harper on 2017-03-22

unittest: block_meta.meta_simple and write_image_to_disk path

410. By Ryan Harper on 2017-03-22

revert smtar changes

411. By Ryan Harper on 2017-03-22

Update supported source types and extractor mappings

412. By Ryan Harper on 2017-03-22

Remove debugging from block_meta

413. By Ryan Harper on 2017-03-22

Rename Snappy->UbuntuCore; Add UbuntuCore curthooks unittets

414. By Ryan Harper on 2017-03-22

unittest: add ubuntu-core-network-config

415. By Ryan Harper on 2017-03-22

unittest: ubuntu-core check remove cloud-init.disabled if present

416. By Ryan Harper on 2017-03-22

unittest: handle_cloudconfig invalid config

417. By Ryan Harper on 2017-03-22

Drop useless S (--sparse) flag on extraction; add unittest for dd-tgz cmdline

418. By Ryan Harper on 2017-03-22

Use 'systemd-data/var/lib/snapd' to confirm target is Ubuntu-Core

419. By Ryan Harper on 2017-03-22

remove added whitespace

420. By Ryan Harper on 2017-03-22

merge from trunk

421. By Ryan Harper on 2017-03-22

vmtest: disable ubuntu-core until images are published in simplestreams

[Scott Moser (smoser)]

just make those changes and i think its good.

review: Approve

[lp:~raharper/curtin/trunk.snappy-dd]

422. By Ryan Harper on 2017-03-22

Address remaining review comments

[Ryan Harper (raharper)]

Applied changes, running vmtest for regression checks here:

https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-debug/2/

=== modified file 'curtin/block/__init__.py'
--- curtin/block/__init__.py	2017-02-08 06:02:02 +0000
+++ curtin/block/__init__.py	2017-03-22 21:04:27 +0000
@@ -540,10 +540,15 @@
     return multipath_wwids
 
 
-def get_root_device(dev, fpath="curtin"):
-    """
-    Get root partition for specified device, based on presence of /curtin.
-    """
+def get_root_device(dev, paths=None):
+    """
+    Get root partition for specified device, based on presence of any
+    paths in the provided paths list:
+    """
+    if paths is None:
+        paths = ["curtin"]
+    LOG.debug('Searching for filesystem on %s containing one of: %s',
+              dev, paths)
     partitions = get_pardevs_on_blockdevs(dev)
     target = None
     tmp_mount = tempfile.mkdtemp()
@@ -553,10 +558,13 @@
         try:
             util.do_mount(dev_path, tmp_mount)
             mp = tmp_mount
-            curtin_dir = os.path.join(tmp_mount, fpath)
-            if os.path.isdir(curtin_dir):
-                target = dev_path
-                break
+            for path in paths:
+                fullpath = os.path.join(tmp_mount, path)
+                if os.path.isdir(fullpath):
+                    target = dev_path
+                    LOG.debug("Found path '%s' on device '%s'",
+                              path, dev_path)
+                    break
         except Exception:
             pass
         finally:
@@ -564,9 +572,10 @@
                 util.do_umount(mp)
 
     os.rmdir(tmp_mount)
-
     if target is None:
-        raise ValueError("Could not find root device")
+        raise ValueError(
+            "Did not find any filesystem on %s that contained one of %s" %
+            (dev, paths))
     return target
 
 

=== modified file 'curtin/commands/block_meta.py'
--- curtin/commands/block_meta.py	2017-03-20 00:10:58 +0000
+++ curtin/commands/block_meta.py	2017-03-22 21:04:27 +0000
@@ -77,14 +77,26 @@
     """
     Write disk image to block device
     """
+    LOG.info('writing image to disk %s, %s', source, dev)
+    extractor = {
+        'dd-tgz': '|tar -xOzf -',
+        'dd-txz': '|tar -xOJf -',
+        'dd-tbz': '|tar -xOjf -',
+        'dd-tar': '|smtar -xOf -',
+        'dd-bz2': '|bzcat',
+        'dd-gz': '|zcat',
+        'dd-xz': '|xzcat',
+        'dd-raw': ''
+    }
     (devname, devnode) = block.get_dev_name_entry(dev)
     util.subp(args=['sh', '-c',
-                    ('wget "$1" --progress=dot:mega -O - |'
-                     'tar -SxOzf - | dd of="$2"'),
-                    '--', source, devnode])
+                    ('wget "$1" --progress=dot:mega -O - ' +
+                     extractor[source['type']] + '| dd bs=4M of="$2"'),
+                    '--', source['uri'], devnode])
     util.subp(['partprobe', devnode])
     udevadm_settle()
-    return block.get_root_device([devname, ])
+    paths = ["curtin", "system-data/var/lib/snapd"]
+    return block.get_root_device([devname], paths=paths)
 
 
 def get_bootpt_cfg(cfg, enabled=False, fstype=None, root_fstype=None):

=== modified file 'curtin/commands/curthooks.py'
--- curtin/commands/curthooks.py	2017-02-11 18:15:51 +0000
+++ curtin/commands/curthooks.py	2017-03-22 21:04:27 +0000
@@ -672,6 +672,83 @@
     util.system_upgrade(target=target)
 
 
+def handle_cloudconfig(cfg, target=None):
+    """write cloud-init configuration files into target
+
+    cloudconfig format is a dictionary of keys and values of content
+
+    cloudconfig:
+      cfg-datasource:
+        content:
+         |
+         #cloud-cfg
+         datasource_list: [ MAAS ]
+      cfg-maas:
+        content:
+         |
+         #cloud-cfg
+         reporting:
+           maas: { consumer_key: 8cW9kadrWZcZvx8uWP,
+                   endpoint: 'http://XXX',
+                   token_key: jD57DB9VJYmDePCRkq,
+                   token_secret: mGFFMk6YFLA3h34QHCv22FjENV8hJkRX,
+                   type: webhook}
+    """
+    # check that cfg is dict
+    if not isinstance(cfg, dict):
+        raise ValueError("cloudconfig configuration is not in dict format")
+
+    # for each item in the dict
+    #   generate a path based on item key
+    #   if path is already in the item, LOG warning, and use generated path
+    for cfgname, cfgvalue in cfg.items():
+        cfgpath = "50-cloudconfig-%s.cfg" % cfgname
+        if 'path' in cfgvalue:
+            LOG.warning("cloudconfig ignoring 'path' key in config")
+        cfgvalue['path'] = cfgpath
+
+    # re-use write_files format and adjust target to prepend
+    LOG.debug('Calling write_files with cloudconfig @ %s', target)
+    LOG.debug('Injecting cloud-config:\n%s', cfg)
+    write_files({'write_files': cfg}, target)
+
+
+def ubuntu_core_curthooks(cfg, target=None):
+    """ Ubuntu-Core 16 images cannot execute standard curthooks
+        Instead we copy in any cloud-init configuration to
+        the 'LABEL=writable' partition mounted at target.
+    """
+
+    ubuntu_core_target = os.path.join(target, "system-data")
+    cc_target = os.path.join(ubuntu_core_target, 'etc/cloud/cloud.cfg.d')
+
+    cloudconfig = cfg.get('cloudconfig', None)
+    if cloudconfig:
+        # remove cloud-init.disabled, if found
+        cloudinit_disable = os.path.join(ubuntu_core_target,
+                                         'etc/cloud/cloud-init.disabled')
+        if os.path.exists(cloudinit_disable):
+            util.del_file(cloudinit_disable)
+
+        handle_cloudconfig(cloudconfig, target=cc_target)
+
+    netconfig = cfg.get('network', None)
+    if netconfig:
+        LOG.info('Writing network configuration')
+        ubuntu_core_netconfig = os.path.join(cc_target,
+                                             "50-network-config.cfg")
+        util.write_file(ubuntu_core_netconfig,
+                        content=config.dump_config(netconfig))
+
+
+def target_is_ubuntu_core(target):
+    """Check if Ubuntu-Core specific directory is present at target"""
+    if target:
+        return os.path.exists(util.target_path(target,
+                                               'system-data/var/lib/snapd'))
+    return False
+
+
 def curthooks(args):
     state = util.load_command_environment()
 
@@ -693,6 +770,14 @@
     cfg = config.load_command_config(args, state)
     stack_prefix = state.get('report_stack_prefix', '')
 
+    if target_is_ubuntu_core(target):
+        LOG.info('Detected Ubuntu-Core image, running hooks')
+        with events.ReportEventStack(
+                name=stack_prefix, reporting_enabled=True, level="INFO",
+                description="Configuring Ubuntu-Core for first boot"):
+            rv = ubuntu_core_curthooks(cfg, target)
+        sys.exit(rv)
+
     with events.ReportEventStack(
             name=stack_prefix + '/writing-config',
             reporting_enabled=True, level="INFO",

=== modified file 'curtin/util.py'
--- curtin/util.py	2017-02-17 21:02:20 +0000
+++ curtin/util.py	2017-03-22 21:04:27 +0000
@@ -829,7 +829,8 @@
     if type(source) is dict:
         # already sanitized?
         return source
-    supported = ['tgz', 'dd-tgz']
+    supported = ['tgz', 'dd-tgz', 'dd-tbz', 'dd-txz', 'dd-tar', 'dd-bz2',
+                 'dd-gz', 'dd-xz', 'dd-raw']
     deftype = 'tgz'
     for i in supported:
         prefix = i + ":"
@@ -852,7 +853,7 @@
         if type(sources[i]) is not dict:
             continue
         if sources[i]['type'].startswith('dd-'):
-            src.append(sources[i]['uri'])
+            src.append(sources[i])
     return src
 
 

=== added file 'examples/tests/ubuntu_core.yaml'
--- examples/tests/ubuntu_core.yaml	1970-01-01 00:00:00 +0000
+++ examples/tests/ubuntu_core.yaml	2017-03-22 21:04:27 +0000
@@ -0,0 +1,13 @@
+# This pushes curtin through a automatic installation
+# where no storage configuration is necessary.
+placeholder_simple_install: unused
+showtrace: true
+cloudconfig:
+  curtin_vmtest:
+    path: /etc/cloud/cloud.cfg.d/99-curtin-vmtest.cfg
+    content:
+      |
+      #cloud-config
+      snappy:
+        email: [e-mail]
+        packages: [hello, part-numpy, part-cython]

=== modified file 'tests/unittests/test_commands_block_meta.py'
--- tests/unittests/test_commands_block_meta.py	2017-02-01 03:01:54 +0000
+++ tests/unittests/test_commands_block_meta.py	2017-03-22 21:04:27 +0000
@@ -1,5 +1,6 @@
 from unittest import TestCase
-from mock import patch
+from mock import patch, call
+from argparse import Namespace
 
 from curtin.commands import block_meta
 
@@ -17,6 +18,108 @@
         setattr(self, attr, p)
 
 
+class TestBlockMetaSimple(BlockMetaTestBase):
+    def setUp(self):
+        super(TestBlockMetaSimple, self).setUp()
+        self.target = "my_target"
+
+        # block_meta
+        basepath = 'curtin.commands.block_meta.'
+        self.add_patch(basepath + 'get_bootpt_cfg', 'mock_bootpt_cfg')
+        self.add_patch(basepath + 'get_partition_format_type',
+                       'mock_part_fmt_type')
+        # block
+        self.add_patch('curtin.block.stop_all_unused_multipath_devices',
+                       'mock_block_stop_mp')
+        self.add_patch('curtin.block.get_installable_blockdevs',
+                       'mock_block_get_installable_bdevs')
+        self.add_patch('curtin.block.get_dev_name_entry',
+                       'mock_block_get_dev_name_entry')
+        self.add_patch('curtin.block.get_root_device',
+                       'mock_block_get_root_device')
+        self.add_patch('curtin.block.is_valid_device',
+                       'mock_block_is_valid_device')
+        # config
+        self.add_patch('curtin.config.load_command_config',
+                       'mock_config_load')
+        # util
+        self.add_patch('curtin.util.subp', 'mock_subp')
+        self.add_patch('curtin.util.load_command_environment',
+                       'mock_load_env')
+
+    def test_write_image_to_disk(self):
+        source = {
+            'type': 'dd-xz',
+            'uri': 'http://myhost/curtin-unittest-dd.xz'
+        }
+        devname = "fakedisk1p1"
+        devnode = "/dev/" + devname
+        self.mock_block_get_dev_name_entry.return_value = (devname, devnode)
+
+        block_meta.write_image_to_disk(source, devname)
+
+        wget = ['sh', '-c',
+                'wget "$1" --progress=dot:mega -O - |xzcat| dd bs=4M of="$2"',
+                '--', source['uri'], devnode]
+        self.mock_block_get_dev_name_entry.assert_called_with(devname)
+        self.mock_subp.assert_has_calls([call(args=wget),
+                                         call(['partprobe', devnode]),
+                                         call(['udevadm', 'settle'])])
+        paths = ["curtin", "system-data/var/lib/snapd"]
+        self.mock_block_get_root_device.assert_called_with([devname],
+                                                           paths=paths)
+
+    def test_write_image_to_disk_ddtgz(self):
+        source = {
+            'type': 'dd-tgz',
+            'uri': 'http://myhost/curtin-unittest-dd.tgz'
+        }
+        devname = "fakedisk1p1"
+        devnode = "/dev/" + devname
+        self.mock_block_get_dev_name_entry.return_value = (devname, devnode)
+
+        block_meta.write_image_to_disk(source, devname)
+
+        wget = ['sh', '-c',
+                'wget "$1" --progress=dot:mega -O - |'
+                'tar -xOzf -| dd bs=4M of="$2"',
+                '--', source['uri'], devnode]
+        self.mock_block_get_dev_name_entry.assert_called_with(devname)
+        self.mock_subp.assert_has_calls([call(args=wget),
+                                         call(['partprobe', devnode]),
+                                         call(['udevadm', 'settle'])])
+        paths = ["curtin", "system-data/var/lib/snapd"]
+        self.mock_block_get_root_device.assert_called_with([devname],
+                                                           paths=paths)
+
+    @patch('curtin.commands.block_meta.write_image_to_disk')
+    def test_meta_simple_calls_write_img(self, mock_write_image):
+        devname = "fakedisk1p1"
+        devnode = "/dev/" + devname
+        sources = {
+            'unittest': {'type': 'dd-xz',
+                         'uri': 'http://myhost/curtin-unittest-dd.xz'}
+        }
+        config = {
+            'block-meta': {'devices': [devname]},
+            'sources': sources,
+        }
+        self.mock_config_load.return_value = config
+        self.mock_load_env.return_value = {'target': self.target}
+        self.mock_block_is_valid_device.return_value = True
+        self.mock_block_get_dev_name_entry.return_value = (devname, devnode)
+        mock_write_image.return_value = devname
+
+        args = Namespace(target=self.target, devices=None, mode=None,
+                         boot_fstype=None, fstype=None)
+
+        block_meta.meta_simple(args)
+
+        mock_write_image.assert_called_with(sources.get('unittest'), devname)
+        self.mock_subp.assert_has_calls(
+            [call(['mount', devname, self.target])])
+
+
 class TestBlockMeta(BlockMetaTestBase):
     def setUp(self):
         super(TestBlockMeta, self).setUp()

=== modified file 'tests/unittests/test_curthooks.py'
--- tests/unittests/test_curthooks.py	2017-02-09 19:59:11 +0000
+++ tests/unittests/test_curthooks.py	2017-03-22 21:04:27 +0000
@@ -6,6 +6,7 @@
 
 from curtin.commands import curthooks
 from curtin import util
+from curtin import config
 from curtin.reporter import events
 
 
@@ -175,4 +176,147 @@
         self.assertEqual([], self.mock_install_packages.call_args_list)
 
 
+class TestUbuntuCoreHooks(CurthooksBase):
+    def setUp(self):
+        super(TestUbuntuCoreHooks, self).setUp()
+        self.target = None
+
+    def tearDown(self):
+        if self.target:
+            shutil.rmtree(self.target)
+
+    def test_target_is_ubuntu_core(self):
+        self.target = tempfile.mkdtemp()
+        ubuntu_core_path = os.path.join(self.target, 'system-data',
+                                        'var/lib/snapd')
+        util.ensure_dir(ubuntu_core_path)
+        self.assertTrue(os.path.isdir(ubuntu_core_path))
+        is_core = curthooks.target_is_ubuntu_core(self.target)
+        self.assertTrue(is_core)
+
+    def test_target_is_ubuntu_core_no_target(self):
+        is_core = curthooks.target_is_ubuntu_core(self.target)
+        self.assertFalse(is_core)
+
+    def test_target_is_ubuntu_core_noncore_target(self):
+        self.target = tempfile.mkdtemp()
+        non_core_path = os.path.join(self.target, 'curtin')
+        util.ensure_dir(non_core_path)
+        self.assertTrue(os.path.isdir(non_core_path))
+        is_core = curthooks.target_is_ubuntu_core(self.target)
+        self.assertFalse(is_core)
+
+    @patch('curtin.util.write_file')
+    @patch('curtin.util.del_file')
+    @patch('curtin.commands.curthooks.handle_cloudconfig')
+    def test_curthooks_no_config(self, mock_handle_cc, mock_del_file,
+                                 mock_write_file):
+        self.target = tempfile.mkdtemp()
+        cfg = {}
+        curthooks.ubuntu_core_curthooks(cfg, target=self.target)
+        self.assertEqual(len(mock_handle_cc.call_args_list), 0)
+        self.assertEqual(len(mock_del_file.call_args_list), 0)
+        self.assertEqual(len(mock_write_file.call_args_list), 0)
+
+    @patch('curtin.commands.curthooks.handle_cloudconfig')
+    def test_curthooks_cloud_config_remove_disabled(self, mock_handle_cc):
+        self.target = tempfile.mkdtemp()
+        uc_cloud = os.path.join(self.target, 'system-data', 'etc/cloud')
+        cc_disabled = os.path.join(uc_cloud, 'cloud-init.disabled')
+        cc_path = os.path.join(uc_cloud, 'cloud.cfg.d')
+
+        util.ensure_dir(uc_cloud)
+        util.write_file(cc_disabled, content="# disable cloud-init\n")
+        cfg = {
+            'cloudconfig': {
+                'file1': {
+                    'content': "Hello World!\n",
+                }
+            }
+        }
+        self.assertTrue(os.path.exists(cc_disabled))
+        curthooks.ubuntu_core_curthooks(cfg, target=self.target)
+
+        mock_handle_cc.assert_called_with(cfg.get('cloudconfig'),
+                                          target=cc_path)
+        self.assertFalse(os.path.exists(cc_disabled))
+
+    @patch('curtin.util.write_file')
+    @patch('curtin.util.del_file')
+    @patch('curtin.commands.curthooks.handle_cloudconfig')
+    def test_curthooks_cloud_config(self, mock_handle_cc, mock_del_file,
+                                    mock_write_file):
+        self.target = tempfile.mkdtemp()
+        cfg = {
+            'cloudconfig': {
+                'file1': {
+                    'content': "Hello World!\n",
+                }
+            }
+        }
+        curthooks.ubuntu_core_curthooks(cfg, target=self.target)
+
+        self.assertEqual(len(mock_del_file.call_args_list), 0)
+        cc_path = os.path.join(self.target,
+                               'system-data/etc/cloud/cloud.cfg.d')
+        mock_handle_cc.assert_called_with(cfg.get('cloudconfig'),
+                                          target=cc_path)
+        self.assertEqual(len(mock_write_file.call_args_list), 0)
+
+    @patch('curtin.util.write_file')
+    @patch('curtin.util.del_file')
+    @patch('curtin.commands.curthooks.handle_cloudconfig')
+    def test_curthooks_net_config(self, mock_handle_cc, mock_del_file,
+                                  mock_write_file):
+        self.target = tempfile.mkdtemp()
+        cfg = {
+            'network': {
+                'version': '1',
+                'config': [{'type': 'physical',
+                            'name': 'eth0', 'subnets': [{'type': 'dhcp4'}]}]
+            }
+        }
+        curthooks.ubuntu_core_curthooks(cfg, target=self.target)
+
+        self.assertEqual(len(mock_del_file.call_args_list), 0)
+        self.assertEqual(len(mock_handle_cc.call_args_list), 0)
+        netcfg_path = os.path.join(self.target,
+                                   'system-data',
+                                   'etc/cloud/cloud.cfg.d',
+                                   '50-network-config.cfg')
+        netcfg = config.dump_config(cfg.get('network'))
+        mock_write_file.assert_called_with(netcfg_path,
+                                           content=netcfg)
+        self.assertEqual(len(mock_del_file.call_args_list), 0)
+
+    @patch('curtin.commands.curthooks.write_files')
+    def test_handle_cloudconfig(self, mock_write_files):
+        cc_target = "tmpXXXX/systemd-data/etc/cloud/cloud.cfg.d"
+        cloudconfig = {
+            'file1': {
+                'content': "Hello World!\n",
+            },
+            'foobar': {
+                'path': '/sys/wark',
+                'content': "Engauge!\n",
+            }
+        }
+
+        expected_cfg = {
+            'write_files': {
+                'file1': {
+                    'path': '50-cloudconfig-file1.cfg',
+                    'content': cloudconfig['file1']['content']},
+                'foobar': {
+                    'path': '50-cloudconfig-foobar.cfg',
+                    'content': cloudconfig['foobar']['content']}
+            }
+        }
+        curthooks.handle_cloudconfig(cloudconfig, target=cc_target)
+        mock_write_files.assert_called_with(expected_cfg, cc_target)
+
+    def test_handle_cloudconfig_bad_config(self):
+        with self.assertRaises(ValueError):
+            curthooks.handle_cloudconfig([], target="foobar")
+
 # vi: ts=4 expandtab syntax=python

=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py	2017-03-01 16:18:42 +0000
+++ tests/vmtests/__init__.py	2017-03-22 21:04:27 +0000
@@ -47,6 +47,9 @@
 
 _TOPDIR = None
 
+UC16_IMAGE = os.path.join(IMAGE_DIR,
+                          'ubuntu-core-16/amd64/20170217/root-image.xz')
+
 
 def remove_empty_dir(dirpath):
     if os.path.exists(dirpath):
@@ -356,6 +359,7 @@
     target_distro = None
     target_release = None
     target_krel = None
+    target_ftype = "vmtest.root-tgz"
 
     def shortDescription(self):
         return None
@@ -375,14 +379,19 @@
 
         # get local absolute filesystem paths for the OS tarball to be
         # installed
-        img_verstr, found = get_images(
-            IMAGE_SRC_URL, IMAGE_DIR,
-            cls.target_distro if cls.target_distro else cls.distro,
-            cls.target_release if cls.target_release else cls.release,
-            cls.arch, krel=cls.target_krel, sync=CURTIN_VMTEST_IMAGE_SYNC,
-            ftypes=('vmtest.root-tgz',))
-        logger.debug("Target Tarball %s\n, ftypes: %s\n", img_verstr, found)
-        logger.info("Target Tarball: %s", img_verstr)
+        if cls.target_ftype == "vmtest.root-tgz":
+            img_verstr, found = get_images(
+                IMAGE_SRC_URL, IMAGE_DIR,
+                cls.target_distro if cls.target_distro else cls.distro,
+                cls.target_release if cls.target_release else cls.release,
+                cls.arch, krel=cls.target_krel, sync=CURTIN_VMTEST_IMAGE_SYNC,
+                ftypes=('vmtest.root-tgz',))
+            logger.debug("Target Tarball %s\n, ftypes: %s\n",
+                         img_verstr, found)
+            logger.info("Target Tarball: %s", img_verstr)
+        else:
+            logger.info('get-testfiles UC16 hack!')
+            found = {'root-image.xz': UC16_IMAGE}
         ftypes.update(found)
         return ftypes
 
@@ -532,8 +541,12 @@
             cmd.extend(["--append=" + cls.extra_kern_args])
 
         # publish the root tarball
-        install_src = "PUBURL/" + os.path.basename(ftypes['vmtest.root-tgz'])
-        cmd.append("--publish=%s" % ftypes['vmtest.root-tgz'])
+        install_src = "PUBURL/" + os.path.basename(ftypes[cls.target_ftype])
+        if cls.target_ftype == 'vmtest.root-tgz':
+            cmd.append("--publish=%s" % ftypes['vmtest.root-tgz'])
+        else:
+            cmd.append("--publish=%s::dd-xz" % ftypes[cls.target_ftype])
+        logger.info("Publishing src as %s", cmd[-1])
 
         # check for network configuration
         cls.network_state = curtin_net.parse_net_config(cls.conf_file)
@@ -1261,8 +1274,9 @@
     failsafe_poweroff = textwrap.dedent("""#!/bin/sh -x
         [ -e /etc/centos-release -o -e /etc/redhat-release ] &&
             { shutdown -P now "Shutting down on centos"; }
-        [ "$(lsb_release -sc)" = "precise" ] &&
-            { shutdown -P now "Shutting down on precise"; }
+        if grep -i -q precise /etc/os-release; then
+            shutdown -P now "Shutting down on precise";
+        fi
         exit 0;
         """)
 

=== modified file 'tests/vmtests/releases.py'
--- tests/vmtests/releases.py	2017-02-01 20:32:13 +0000
+++ tests/vmtests/releases.py	2017-03-22 21:04:27 +0000
@@ -15,6 +15,11 @@
     target_distro = "centos"
 
 
+class _UbuntuCoreUbuntuBase(_UbuntuBase):
+    # base for installing UbuntuCore root-image.xz from ubuntu base
+    target_distro = "ubuntu-core-16"
+
+
 class _Centos70FromXenialBase(_CentosFromUbuntuBase):
     # release for boot
     release = "xenial"
@@ -22,6 +27,13 @@
     target_release = "centos70"
 
 
+class _UbuntuCore16FromXenialBase(_UbuntuCoreUbuntuBase):
+    # release for boot
+    release = "xenial"
+    # release for target
+    target_release = "ubuntu-core-16"
+
+
 class _Centos66FromXenialBase(_CentosFromUbuntuBase):
     release = "xenial"
     target_release = "centos66"
@@ -106,7 +118,12 @@
     centos66fromxenial = _Centos66FromXenialBase
 
 
+class _UbuntuCoreReleases(object):
+    uc16fromxenial = _UbuntuCore16FromXenialBase
+
+
 base_vm_classes = _Releases
 centos_base_vm_classes = _CentosReleases
+ubuntu_core_base_vm_classes = _UbuntuCoreReleases
 
 # vi: ts=4 expandtab syntax=python

=== added file 'tests/vmtests/test_ubuntu_core.py'
--- tests/vmtests/test_ubuntu_core.py	1970-01-01 00:00:00 +0000
+++ tests/vmtests/test_ubuntu_core.py	2017-03-22 21:04:27 +0000
@@ -0,0 +1,46 @@
+from . import VMBaseClass
+from .releases import ubuntu_core_base_vm_classes as relbase
+
+import textwrap
+
+
+class TestUbuntuCoreAbs(VMBaseClass):
+    target_ftype = "root-image.xz"
+    interactive = False
+    conf_file = "examples/tests/ubuntu_core.yaml"
+    collect_scripts = [textwrap.dedent("""
+        cd OUTPUT_COLLECT_D
+        cat /proc/partitions > proc_partitions
+        ls -al /dev/disk/by-uuid/ > ls_uuid
+        cat /etc/fstab > fstab
+        find /etc/network/interfaces.d > find_interfacesd
+        snap list > snap_list
+        cp -a /run/cloud-init ./run_cloud_init |:
+        cp -a /etc/cloud ./etc_cloud |:
+        cp -a /home . |:
+        cp -a /var/lib/extrausers . |:
+        """)]
+
+    def test_output_files_exist(self):
+        self.output_files_exist(["snap_list"])
+
+    def test_ubuntu_core_snaps_installed(self):
+        snap_list = self.load_collect_file('snap_list')
+        print(snap_list)
+        for snap in ['core', 'pc', 'pc-kernel', 'hello',
+                     'part-cython', 'part-numpy']:
+            print('check for "%s"' % snap)
+            self.assertIn(snap, snap_list)
+
+    def test_ubuntu_core_extrausers(self):
+        extrausers_passwd = self.load_collect_file('extrausers/passwd')
+        self.assertIn('ubuntu', extrausers_passwd)
+
+    def test_ubuntu_core_ds_identify(self):
+        run_ci_config = self.load_collect_file('run_cloud_init/cloud.cfg')
+        expected_config = "datasource_list: [ NoCloud, None ]\n"
+        self.assertEqual(expected_config, run_ci_config)
+
+
+class UbuntuCore16TestUbuntuCore(relbase.uc16fromxenial, TestUbuntuCoreAbs):
+    __test__ = False

=== modified file 'tools/launch'
--- tools/launch	2017-02-17 21:36:49 +0000
+++ tools/launch	2017-03-22 21:04:27 +0000
@@ -60,7 +60,7 @@
 
    Example:
     * boot myboot.img, and install my-root.tar.gz
-      ${0##*/} myboot.img --publish my-root.tar.gz curtin \
+      ${0##*/} myboot.img --publish my-root.tar.gz curtin \\
          install PUBURL/my-root.tar.gz
 
 EOF
@@ -563,18 +563,20 @@
     ip=${_RET}
 
     local tok src pub fpath
-    # tok in pubs looks like file[:pubname]
+    # tok in pubs looks like file[:pubname][:stype]
     # link them into the temp dir for publishing
     for tok in "${pubs[@]}"; do
-        case "$tok" in
-            *:*) src="${tok%:*}"; pub="${tok##*:}";;
-            *) src=${tok}; pub="";;
-        esac
+        src=$(echo "${tok}" | awk -F: '{print $1}')
+        pub=$(echo "${tok}" | awk -F: '{print $2}')
+        stype=$(echo "${tok}" | awk -F: '{print $3}')
         fpath=$(readlink -f "$src") ||
             { error "'$src': failed to get path"; return 1; }
-        if [ -n "$pub" ]; then
+        if [ -z "$pub" ]; then
             pub="${src##*/}"
         fi
+        if [ -z "$stype" ]; then
+            stype="tgz"
+        fi
         ln -sf "$fpath" "${TEMP_D}/${pub}"
     done
 
@@ -582,9 +584,10 @@
         { error "failed to start http service"; return 1; }
     http_port=$_RET
     burl="http://$ip:${http_port}/"
-    # now replace PUBURL anywhere in cmdargs
+    # now replace PUBURL anywhere in cmdargs, and prefix with
+    # source type (tgz or dd-tgz)
     for((i=0;i<${#cmdargs[@]};i++)); do
-        cmdargs[$i]=${cmdargs[$i]//PUBURL/$burl}
+        cmdargs[$i]=${cmdargs[$i]//PUBURL/${stype}:$burl}
     done
 
     local addargs="" f=""