Merge lp:~raharper/curtin/trunk.power8 into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

autogenerate vmtest data collection

All vmtest classes define lots of duplicate data and require knowledge of the
contents of the input config yaml. This is cumbersome and error-prone.
Fix this by auto-generating the scripts to collect data based on the input
config.

Updated block_meta's get_path_to_storage_volume:
  - change suffix to no_sync and drop the device sync
  - add wrapper function for existing callers which calls devsync
  - implement handlers for 'format', 'mount', 'lvm_partition' types

Updated block_meta's determine_partition_kname to handle devices with
wwn assigned.

Refactored basic vmtest to read the parsed storage config and calculate
the correct expected files collected and then validate internal data.

This currently passes for XenialBasic.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:383
https://server-team-jenkins.canonical.com/job/curtin-ci/286/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/283/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/286/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.power8]

384. By Ryan Harper on 2016-06-29

merge from trunk, resolv conflicts

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:384
https://server-team-jenkins.canonical.com/job/curtin-ci/287/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/284/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/287/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

why the funny '_no_sync' name ? versus a 'sync=False' argument?

i'm not opposed to the overall injection of prep partition.
i dont understand yet how you check the exported data though.

[Ryan Harper (raharper)]

On Wed, Jun 29, 2016 at 1:54 PM, Scott Moser <email address hidden> wrote:

> why the funny '_no_sync' name ? versus a 'sync=False' argument?
>

because class instantiation. I didn't want to touch everywhere else that
calls it to set sync=True during a real install. If I set it to True by
default
then it attempts to devsync when I just want it to generate plausible paths
at startup time.

>
> i'm not opposed to the overall injection of prep partition.
> i dont understand yet how you check the exported data though.
>

We're programtically generating blkid and parted commands and
writing them at known paths based on a unique disk identifier (wwn)

>
> Diff comments:
>
> > === modified file 'curtin/commands/block_meta.py'
> > --- curtin/commands/block_meta.py 2016-06-08 21:58:29 +0000
> > +++ curtin/commands/block_meta.py 2016-06-29 18:30:49 +0000
> > @@ -266,10 +266,20 @@
> >
> >
> > def determine_partition_kname(disk_kname, partition_number):
> > - for dev_type in ["nvme", "mmcblk"]:
> > - if disk_kname.startswith(dev_type):
> > - partition_number = "p%s" % partition_number
> > - break
> > + """ Modify, if necessary, the prefix of a partition
> > + depending on the type of disk kernel name
>
> maybe the function name was always wrong, but now it is wrong for sure.
> wwn-0xff-part1 is not a kname (kernel name).
>

meh, we can drop the k, determine_partition_name

>
> > +
> > + (sda, 2) => sda2
> > + (nvmen1, 1) => nvmen1p1
> > + (mmcblk0, 5) => mmcblk0p5
> > + (wwn-0xfff, 1) => wwn-0xfff-part1
> > + (scsi-$ID, 2) => scsi-$ID-part2
> > + """
> > + if disk_kname.startswith("nvme") or disk_kname.startswith("mmcblk"):
> > + partition_number = "p%s" % partition_number
> > + elif disk_kname.startswith("wwn-") or
> disk_kname.startswith('scsi-'):
> > + partition_number = "-part%s" % partition_number
> > +
> > return "%s%s" % (disk_kname, partition_number)
> >
> >
> > @@ -653,6 +687,8 @@
> > info.get('id'), device, disk_ptable)
> > LOG.debug("partnum: %s offset_sectors: %s length_sectors: %s",
> > partnumber, offset_sectors, length_sectors)
> > + (out, err) = util.subp(["cat", "/proc/partitions"], capture=True)
> > + LOG.debug("partitions:\n%s", out)
>
> util.load_file("/proc/partitions") ?
>
> debugging, will drop, but good use of load_file.

> > if disk_ptable == "msdos":
> > if flag in ["extended", "logical", "primary"]:
> > partition_type = flag
> > @@ -662,6 +698,15 @@
> > "%ss" % offset_sectors, "%ss" % str(offset_sectors +
> > length_sectors)]
> > util.subp(cmd, capture=True)
> > +
> > + (out, err) = util.subp(["cat", "/proc/partitions"],
> capture=True)
> > + LOG.debug("partitions:\n%s", out)
>
> i guess its just debug. but load_file just as easy.
>
> will drop

> > +
> > + if flag in ["prep"]:
> > + cmd = ["parted", "--script", disk, "set %d boot on" %
> partnumber]
> > + util.subp(cmd, capture=True)
> > + cmd = ["parted", "--script"...

[Scott Moser (smoser)]

we really should have some tests that show full use of a disk, such that the inserted partition would not fit.
thanks for the reply. generally i think this does look nice. over time i hope to have less of the copy'd stuf fall round in vmtest and more stuff like this is doing.

[lp:~raharper/curtin/trunk.power8]

385. By Ryan Harper on 2016-06-30

merge from diglett fixing lvm and common vmtests

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:385
https://server-team-jenkins.canonical.com/job/curtin-ci/291/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/288/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/291/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.power8]

386. By Ryan Harper on 2016-07-06

merge from diglett and mdadm-bcache fixes

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:386
https://server-team-jenkins.canonical.com/job/curtin-ci/296/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/293/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/296/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.power8]

387. By Ryan Harper on 2016-07-08

merge from diglett fixing remaining storage tests.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:387
https://server-team-jenkins.canonical.com/job/curtin-ci/299/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/296/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/299/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.power8]

388. By Ryan Harper on 2016-07-08

merge from diglett, fixing test_simple

389. By Ryan Harper on 2016-07-11

merge from diglett, full tests run fixes

390. By Ryan Harper on 2016-07-19

more fixups from run on power hardware

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:390
https://server-team-jenkins.canonical.com/job/curtin-ci/317/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/329/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/317/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:390
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/21/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/330/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/21/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:390
https://jenkins.ubuntu.com/server/job/curtin-ci/399/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/399
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/399
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/399
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/399
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/399

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/399/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:390
https://jenkins.ubuntu.com/server/job/curtin-ci/416/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/416
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/416
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/416
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/416
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/416

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/416/rebuild

review: Approve (continuous-integration)

Unmerged revisions

390. By Ryan Harper on 2016-07-19

more fixups from run on power hardware

389. By Ryan Harper on 2016-07-11

merge from diglett, full tests run fixes

388. By Ryan Harper on 2016-07-08

merge from diglett, fixing test_simple

387. By Ryan Harper on 2016-07-08

merge from diglett fixing remaining storage tests.

386. By Ryan Harper on 2016-07-06

merge from diglett and mdadm-bcache fixes

385. By Ryan Harper on 2016-06-30

merge from diglett fixing lvm and common vmtests

384. By Ryan Harper on 2016-06-29

merge from trunk, resolv conflicts

383. By Ryan Harper on 2016-06-29

merge from run on diamond/p8 box

382. By Ryan Harper on 2016-06-29

merge from diglett x86 test run

381. By Ryan Harper on 2016-06-28

merge from trunk

=== modified file 'curtin/block/__init__.py'
--- curtin/block/__init__.py	2016-06-10 14:49:22 +0000
+++ curtin/block/__init__.py	2016-07-19 15:04:36 +0000
@@ -436,6 +436,19 @@
     for line in out.splitlines():
         if "UUID" in line:
             return line.split('=')[-1]
+
+    # trusty and precise might not include UUID= output, try via by-uuid
+    (out, _err) = util.subp(["ls", "-al", "/dev/disk/by-uuid"], capture=True)
+    if path in out:
+        # prepend ../../ path
+        kname_base = "../../" + os.path.basename(path)
+        # extract the mapping
+        # <uuid> -> ../../<kernel basename>
+        uuid_list = out.split()
+        path_idx = uuid_list.index(kname_base)
+        uuid = uuid_list[path_idx - 2]
+        return uuid
+
     return ''
 
 
@@ -479,9 +492,10 @@
     serial_udev = serial.replace(' ', '_')
     LOG.info('Processing serial %s via udev to %s', serial, serial_udev)
 
-    disks = list(filter(lambda x: serial_udev in x,
-                        os.listdir("/dev/disk/by-id/")))
+    by_id = os.listdir("/dev/disk/by-id/")
+    disks = list(filter(lambda x: serial_udev in x, by_id))
     if not disks or len(disks) < 1:
+        LOG.debug('cant find serial in: %s', by_id)
         raise ValueError("no disk with serial '%s' found" % serial_udev)
 
     # Sort by length and take the shortest path name, as the longer path names

=== modified file 'curtin/commands/block_meta.py'
--- curtin/commands/block_meta.py	2016-06-08 21:58:29 +0000
+++ curtin/commands/block_meta.py	2016-07-19 15:04:36 +0000
@@ -266,10 +266,20 @@
 
 
 def determine_partition_kname(disk_kname, partition_number):
-    for dev_type in ["nvme", "mmcblk"]:
-        if disk_kname.startswith(dev_type):
-            partition_number = "p%s" % partition_number
-            break
+    """ Modify, if necessary, the prefix of a partition
+        depending on the type of disk kernel name
+
+        (sda, 2) => sda2
+        (nvmen1, 1) => nvmen1p1
+        (mmcblk0, 5) => mmcblk0p5
+        (wwn-0xfff, 1) => wwn-0xfff-part1
+        (scsi-$ID, 2) => scsi-$ID-part2
+    """
+    if disk_kname.startswith("nvme") or disk_kname.startswith("mmcblk"):
+        partition_number = "p%s" % partition_number
+    elif disk_kname.startswith("wwn-") or disk_kname.startswith('scsi-'):
+        partition_number = "-part%s" % partition_number
+
     return "%s%s" % (disk_kname, partition_number)
 
 
@@ -354,11 +364,17 @@
 
 
 def get_path_to_storage_volume(volume, storage_config):
+    """ common path syncs the device path """
+    volpath = get_path_to_storage_volume_no_sync(volume, storage_config)
+    devsync(volpath)
+    return volpath
+
+
+def get_path_to_storage_volume_no_sync(volume, storage_config):
     # Get path to block device for volume. Volume param should refer to id of
     # volume in storage config
 
     LOG.debug('get_path_to_storage_volume for volume {}'.format(volume))
-    devsync_vol = None
     vol = storage_config.get(volume)
     if not vol:
         raise ValueError("volume with id '%s' not found" % volume)
@@ -366,12 +382,11 @@
     # Find path to block device
     if vol.get('type') == "partition":
         partnumber = determine_partition_number(vol.get('id'), storage_config)
-        disk_block_path = get_path_to_storage_volume(vol.get('device'),
-                                                     storage_config)
+        disk_block_path = get_path_to_storage_volume_no_sync(vol.get('device'),
+                                                             storage_config)
         (base_path, disk_kname) = os.path.split(disk_block_path)
         partition_kname = determine_partition_kname(disk_kname, partnumber)
         volume_path = os.path.join(base_path, partition_kname)
-        devsync_vol = os.path.join(disk_block_path)
 
     elif vol.get('type') == "disk":
         # Get path to block device for disk. Device_id param should refer
@@ -400,6 +415,13 @@
         volume_path = os.path.join("/dev/", volgroup.get('name'),
                                    vol.get('name'))
 
+    elif vol.get('type') == "lvm_volgroup":
+        vg_name = vol.get('name')
+        if not vg_name:
+            raise ValueError("lvm_volgroup missing 'name' value in "
+                             "storage config")
+        volume_path = os.path.realpath(os.path.join("/dev/", vg_name))
+
     elif vol.get('type') == "dm_crypt":
         # For dm_crypted partitions, unencrypted block device is at
         # /dev/mapper/<dm_name>
@@ -419,26 +441,38 @@
         # block devs are in the slaves dir there. Then, those blockdevs can be
         # checked against the kname of the devs in the config for the desired
         # bcache device. This is not very elegant though
-        backing_device_kname = os.path.split(get_path_to_storage_volume(
-            vol.get('backing_device'), storage_config))[-1]
-        sys_path = list(filter(lambda x: backing_device_kname in x,
-                               glob.glob("/sys/block/bcache*/slaves/*")))[0]
-        while "bcache" not in os.path.split(sys_path)[-1]:
-            sys_path = os.path.split(sys_path)[0]
-        volume_path = os.path.join("/dev", os.path.split(sys_path)[-1])
+        backing_device_kname = os.path.split(
+            get_path_to_storage_volume_no_sync(vol.get('backing_device'),
+                                               storage_config))[-1]
+        bcache_kdevs = list(filter(lambda x: backing_device_kname in x,
+                                   glob.glob("/sys/block/bcache*/slaves/*")))
+        if len(bcache_kdevs) > 0:
+            sys_path = bcache_kdevs[0]
+            while "bcache" not in os.path.split(sys_path)[-1]:
+                sys_path = os.path.split(sys_path)[0]
+            volume_path = os.path.join("/dev", os.path.split(sys_path)[-1])
+        else:
+            # find all bcache configs present in storage config
+            # and use the order of creation to guess the bcache dev id
+            bcache_devs = [id for (id, val) in storage_config.items()
+                           if val['type'] == 'bcache']
+            bcache_idx = bcache_devs.index(vol.get('id'))
+            volume_path = "/dev/bcache%d" % bcache_idx
         LOG.debug('got bcache volume path {}'.format(volume_path))
 
+    elif vol.get('type') == "format":
+        volume_path = get_path_to_storage_volume_no_sync(vol.get('volume'),
+                                                         storage_config)
+
+    elif vol.get('type') == "mount":
+        volume_path = get_path_to_storage_volume_no_sync(vol.get('device'),
+                                                         storage_config)
     else:
         raise NotImplementedError("cannot determine the path to storage \
             volume '%s' with type '%s'" % (volume, vol.get('type')))
 
-    # sync devices
-    if not devsync_vol:
-        devsync_vol = volume_path
-    devsync(devsync_vol)
-
     LOG.debug('return volume path {}'.format(volume_path))
-    return volume_path
+    return os.path.realpath(volume_path)
 
 
 def disk_handler(info, storage_config):
@@ -653,6 +687,8 @@
              info.get('id'), device, disk_ptable)
     LOG.debug("partnum: %s offset_sectors: %s length_sectors: %s",
               partnumber, offset_sectors, length_sectors)
+    (out, err) = util.subp(["cat", "/proc/partitions"], capture=True)
+    LOG.debug("partitions:\n%s", out)
     if disk_ptable == "msdos":
         if flag in ["extended", "logical", "primary"]:
             partition_type = flag
@@ -662,6 +698,15 @@
                "%ss" % offset_sectors, "%ss" % str(offset_sectors +
                                                    length_sectors)]
         util.subp(cmd, capture=True)
+
+        (out, err) = util.subp(["cat", "/proc/partitions"], capture=True)
+        LOG.debug("partitions:\n%s", out)
+
+        if flag in ["prep"]:
+            cmd = ["parted", "--script", disk, "set %d boot on" % partnumber]
+            util.subp(cmd, capture=True)
+            cmd = ["parted", "--script", disk, "set %d prep on" % partnumber]
+            util.subp(cmd, capture=True)
     elif disk_ptable == "gpt":
         if flag and flag in sgdisk_flags:
             typecode = sgdisk_flags[flag]
@@ -670,6 +715,12 @@
         cmd = ["sgdisk", "--new", "%s:%s:%s" % (partnumber, offset_sectors,
                length_sectors + offset_sectors),
                "--typecode=%s:%s" % (partnumber, typecode), disk]
+        # Power8 prep partitions must have
+        #  GUID=9e1a2d38-c612-4316-aa26-8b49521e5a8b
+        if flag == 'prep':
+            p8guid = '9e1a2d38-c612-4316-aa26-8b49521e5a8b'
+            cmd += ['--partition-guid=%s:%s' % (partnumber, p8guid)]
+
         util.subp(cmd, capture=True)
     else:
         raise ValueError("parent partition has invalid partition table")
@@ -730,11 +781,11 @@
     # Add volume to fstab
     if state['fstab']:
         with open(state['fstab'], "a") as fp:
-            if volume.get('type') in ["raid", "bcache",
-                                      "disk", "lvm_partition"]:
+            if volume.get('type') in ["lvm_partition"]:
                 location = get_path_to_storage_volume(volume.get('id'),
                                                       storage_config)
-            elif volume.get('type') in ["partition", "dm_crypt"]:
+            elif volume.get('type') in ["disk", "raid", "bcache",
+                                        "partition", "dm_crypt"]:
                 location = "UUID=%s" % block.get_volume_uuid(volume_path)
             else:
                 raise ValueError("cannot write fstab for volume type '%s'" %

=== modified file 'curtin/commands/curthooks.py'
--- curtin/commands/curthooks.py	2016-06-24 19:27:52 +0000
+++ curtin/commands/curthooks.py	2016-07-19 15:04:36 +0000
@@ -495,7 +495,8 @@
                 LOG.debug("NOT enabling UEFI nvram updates")
                 LOG.debug("Target system may not boot")
         args.append(target)
-        util.subp(args + instdevs, env=env)
+        LOG.debug('install-grub: %s', args)
+        util.subp(args + instdevs, env=env, capture=True)
 
 
 def update_initramfs(target, all_kernels=False):

=== modified file 'examples/tests/allindata.yaml'
--- examples/tests/allindata.yaml	2016-04-14 01:45:54 +0000
+++ examples/tests/allindata.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdb
+       wwn: '0x0b38330401f57cb6'
        name: main_disk
        grub_device: 1
      - id: bios_boot_partition
@@ -40,11 +40,16 @@
        size: 3GB
        device: sda
        number: 6  # XXX: we really need to stop using id with DiskPartnum
+     - id: sda6        # this partition disables cloud-init's growpart which 
+       type: partition # was breaking the partition size test-case.
+       size: 2GB
+       device: sda
+       number: 7  
      - id: sdb
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdc
+       wwn: '0x0409203126fe097c'
        name: second_disk
      - id: sdb1
        type: partition
@@ -66,7 +71,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdd
+       wwn: '0x5c1c48aa32255436'
        name: third_disk
      - id: sdc1
        type: partition
@@ -88,7 +93,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vde
+       wwn: '0x28984e887e4426c8'
        name: fourth_disk
      - id: sdd1
        type: partition

=== modified file 'examples/tests/basic.yaml'
--- examples/tests/basic.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/basic.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
         type: disk
         ptable: msdos
         model: QEMU HARDDISK
-        path: /dev/vdb
+        wwn: '0x2b873dea52e628f8'
         name: main_disk
         wipe: superblock
         grub_device: true
@@ -39,12 +39,12 @@
         device: sda2_home
       - id: sparedisk_id
         type: disk
-        path: /dev/vdc
+        wwn: '0x4c5a435e6aff3e28'
         name: sparedisk
         wipe: superblock
       - id: btrfs_disk_id
         type: disk
-        path: /dev/vdd
+        wwn: '0x74464deb314c182b'
         name: btrfs_volume
         wipe: superblock
       - id: btrfs_disk_fmt_id
@@ -57,7 +57,7 @@
         device: btrfs_disk_fmt_id
       - id: pnum_disk
         type: disk
-        path: /dev/vde
+        wwn: '0x5b092fb53f5e5816'
         name: pnum_disk
         wipe: superblock
         ptable: gpt

=== modified file 'examples/tests/bcache_basic.yaml'
--- examples/tests/bcache_basic.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/bcache_basic.yaml	2016-07-19 15:04:36 +0000
@@ -4,14 +4,14 @@
   - id: id_rotary0
     type: disk
     name: rotary0
-    path: /dev/vdb
+    wwn: '0x116d1a473ea34188'
     ptable: msdos
     wipe: superblock
     grub_device: true
   - id: id_ssd0
     type: disk
     name: ssd0
-    path: /dev/vdc
+    wwn: '0x594c150c789537ea'
     wipe: superblock
   - id: id_rotary0_part1
     type: partition

=== modified file 'examples/tests/lvm.yaml'
--- examples/tests/lvm.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/lvm.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
         type: disk
         ptable: msdos
         model: QEMU HARDDISK
-        path: /dev/vdb
+        wwn: '0x770f31415cf36ac3'
         name: main_disk
       - id: sda1
         type: partition

=== modified file 'examples/tests/mdadm_bcache.yaml'
--- examples/tests/mdadm_bcache.yaml	2016-04-17 22:35:20 +0000
+++ examples/tests/mdadm_bcache.yaml	2016-07-19 15:04:36 +0000
@@ -7,7 +7,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdb
+       wwn: '0x6f1d6983311e4c9d'
        name: main_disk
      - id: bios_boot_partition
        type: partition
@@ -54,13 +54,13 @@
      - id: sdb
        type: disk
        model: QEMU HARDDISK
-       path: /dev/vdc
+       wwn: '0x11c342997b026e09'
        name: second_disk
      - id: sdc
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdd
+       wwn: '0x14ae3095379d3350'
        name: third_disk
      - id: sdc1
        type: partition

=== modified file 'examples/tests/mdadm_bcache_complex.yaml'
--- examples/tests/mdadm_bcache_complex.yaml	2015-11-25 13:45:53 +0000
+++ examples/tests/mdadm_bcache_complex.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdb
+       wwn: '0x28c124ad5eb83f0a'
        name: main_disk
      - id: bios_boot_partition
        type: partition
@@ -44,13 +44,13 @@
      - id: sdb
        type: disk
        model: QEMU HARDDISK
-       path: /dev/vdc
+       wwn: '0x605e49ec47bd5b38'
        name: second_disk
      - id: sdc
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdd
+       wwn: '0x58521e6a73165fc8'
        name: third_disk
      - id: sdc1
        type: partition

=== modified file 'examples/tests/mirrorboot.yaml'
--- examples/tests/mirrorboot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/mirrorboot.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdb
+       wwn: '0x3ffd076474f457a2'
        name: main_disk
        grub_device: 1
      - id: bios_boot_partition
@@ -22,7 +22,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdc
+       wwn: '0x075a4ceb60d0299f'
        name: second_disk
      - id: sdb1
        type: partition

=== modified file 'examples/tests/nvme.yaml'
--- examples/tests/nvme.yaml	2016-04-04 18:20:33 +0000
+++ examples/tests/nvme.yaml	2016-07-19 15:04:36 +0000
@@ -5,7 +5,7 @@
       - id: main_disk
         type: disk
         ptable: gpt
-        path: /dev/vdb
+        serial: 'IPR-0 1234567890'
         name: main_disk
         wipe: superblock
         grub_device: true
@@ -44,9 +44,9 @@
         device: main_disk_home
       - id: nvme_disk
         type: disk
-        path: /dev/nvme0n1
-        name: nvme_disk
-        wipe: superblock
+        path: /dev/nvme0n1  # nvme's require a serial but the serial isn't
+        name: nvme_disk     # exported into udev until kernel > 4.4 so we 
+        wipe: superblock    # use a path to find the device
         ptable: gpt
       - id: nvme_disk_p1
         type: partition
@@ -63,10 +63,10 @@
         device: nvme_disk
       - id: nvme_disk2
         type: disk
-        path: /dev/nvme1n1
-        wipe: superblock
+        path: /dev/nvme1n1  # nvme's require a serial but the serial isn't
+        name: second_nvme   # exported into udev until kernel > 4.4 so we 
+        wipe: superblock    # use a path to find the device
         ptable: msdos
-        name: second_nvme
       - id: nvme_disk2_p1
         type: partition
         size: 1GB

=== modified file 'examples/tests/raid10boot.yaml'
--- examples/tests/raid10boot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/raid10boot.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdb
+       wwn: '0x21557fcd125b2cb7'
        name: main_disk
        grub_device: 1
      - id: bios_boot_partition
@@ -22,7 +22,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdc
+       wwn: '0x7ade250975512b10'
        name: second_disk
      - id: sdb1
        type: partition
@@ -32,7 +32,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdd
+       wwn: '0x2e902b105c4f6724'
        name: third_disk
      - id: sdc1
        type: partition
@@ -42,7 +42,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vde
+       wwn: '0x5bbd663336493174'
        name: fourth_disk
      - id: sdd1
        type: partition

=== modified file 'examples/tests/raid5bcache.yaml'
--- examples/tests/raid5bcache.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/raid5bcache.yaml	2016-07-19 15:04:36 +0000
@@ -6,31 +6,31 @@
     model: QEMU HARDDISK
     name: sda
     ptable: msdos
-    path: /dev/vdb
+    wwn: '0x603022fd2a1204a3'
     type: disk
     wipe: superblock
   - id: sdb
     model: QEMU HARDDISK
     name: sdb
-    path: /dev/vdc
+    wwn: '0x6ef079fd51bc1aa3'
     type: disk
     wipe: superblock
   - id: sdc
     model: QEMU HARDDISK
     name: sdc
-    path: /dev/vdd
+    wwn: '0x43ac532e78e70785'
     type: disk
     wipe: superblock
   - id: sdd
     model: QEMU HARDDISK
     name: sdd
-    path: /dev/vde
+    wwn: '0x289f09306d0d11a7'
     type: disk
     wipe: superblock
   - id: sde
     model: QEMU HARDDISK
     name: sde
-    path: /dev/vdf
+    wwn: '0x0e1a6948357d0b01'
     type: disk
     wipe: superblock
   - devices:

=== modified file 'examples/tests/raid5boot.yaml'
--- examples/tests/raid5boot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/raid5boot.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdb
+       wwn: '0x40de6e7235387803'
        name: main_disk
        grub_device: 1
      - id: bios_boot_partition
@@ -22,7 +22,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdc
+       wwn: '0x11c93eb225551234'
        name: second_disk
      - id: sdb1
        type: partition
@@ -32,7 +32,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdd
+       wwn: '0x3878752f02f41fc5'
        name: third_disk
      - id: sdc1
        type: partition

=== modified file 'examples/tests/raid6boot.yaml'
--- examples/tests/raid6boot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/raid6boot.yaml	2016-07-19 15:04:36 +0000
@@ -6,7 +6,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdb
+       wwn: '0x2b2518ab3256382e'
        name: main_disk
        grub_device: 1
      - id: bios_boot_partition
@@ -22,7 +22,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdc
+       wwn: '0x4a6a081952db4963'
        name: second_disk
      - id: sdb1
        type: partition
@@ -32,7 +32,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vdd
+       wwn: '0x7e6e04e903cf42e9'
        name: third_disk
      - id: sdc1
        type: partition
@@ -42,7 +42,7 @@
        type: disk
        ptable: gpt
        model: QEMU HARDDISK
-       path: /dev/vde
+       wwn: '0x7fa0135968db1f54'
        name: fourth_disk
      - id: sdd1
        type: partition

=== modified file 'examples/tests/uefi_basic.yaml'
--- examples/tests/uefi_basic.yaml	2016-04-07 21:03:58 +0000
+++ examples/tests/uefi_basic.yaml	2016-07-19 15:04:36 +0000
@@ -4,7 +4,7 @@
   - id: id_disk0
     type: disk
     name: main_disk
-    path: /dev/vdb
+    wwn: '0x3fd832a52c877c49'
     ptable: gpt
     wipe: superblock
     grub_device: true
@@ -22,6 +22,12 @@
     size: 3G
     type: partition
     wipe: superblock
+  - device: id_disk0
+    id: id_disk0_part3
+    number: 3
+    size: 2G
+    type: partition
+    wipe: superblock
   - fstype: fat32
     id: id_efi_format
     label: efi

=== modified file 'examples/tests/vlan_network.yaml'
--- examples/tests/vlan_network.yaml	2016-05-17 20:04:00 +0000
+++ examples/tests/vlan_network.yaml	2016-07-19 15:04:36 +0000
@@ -8,6 +8,8 @@
     - address: 10.245.168.16/21
       dns_nameservers:
       - 10.245.168.2
+      dns_search:
+      - dellstack
       gateway: 10.245.168.1
       type: static
     type: physical

=== modified file 'helpers/common'
--- helpers/common	2016-04-06 19:49:42 +0000
+++ helpers/common	2016-07-19 15:04:36 +0000
@@ -707,6 +707,8 @@
         chroot "$mp" env DEBIAN_FRONTEND=noninteractive sh -ec '
             pkg=$1; shift;
             dpkg-reconfigure "$pkg"
+            initrds=$(ls /boot/initrd.img* || true)
+            [ -z "$initrds" ] && update-initramfs -u
             update-grub
             for d in "$@"; do grub-install "$d" || exit; done' \
             -- "${grub_name}" "${grubdevs[@]}" </dev/null ||

=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py	2016-06-23 14:26:39 +0000
+++ tests/vmtests/__init__.py	2016-07-19 15:04:36 +0000
@@ -15,6 +15,8 @@
 import curtin.net as curtin_net
 import curtin.util as util
 
+from curtin.commands.block_meta import (extract_storage_ordered_dict,
+                                        get_path_to_storage_volume_no_sync)
 from curtin.commands.install import INSTALL_PASS_MSG
 
 from .image_sync import query as imagesync_query
@@ -348,6 +350,111 @@
                               stdout=DEVNULL, stderr=subprocess.STDOUT)
 
 
+def generate_storage_scripts(storage_config):
+    bcache_fs = "ls /sys/fs/bcache > {collectd}/ls_bcache"
+    bcache_cache_mode_map = textwrap.dedent("""
+        for bfile in $(find /sys/class/block/bcache*/bcache); do
+            bcachedev=$(basename `dirname $bfile`)
+            cache_mode=$(cat $bfile/cache_mode)
+            echo $bcachedev:$cache_mode >> {collectd}/bcache_cache_mode_map
+        done""")
+    bcache_super_cache = ("bcache-super-show {cachedev} "
+                          "> {collectd}/bcache_super_cache_{cachebase}")
+    bcache_super_back = ("bcache-super-show {backdev} "
+                         "> {collectd}/bcache_super_backing_{backbase}")
+    bcache_backing_map = textwrap.dedent("""
+        for bfile in $(find /sys/class/block/bcache*/bcache); do
+            bcachedev=$(basename `dirname $bfile`)
+            cfile=`readlink $bfile`
+            cdev=$(basename `dirname $cfile`)
+            echo $bcachedev $cdev >> {collectd}/bcache_backing_map
+        done""")
+    bcache_blkid = ("blkid -o export {devpath} > "
+                    "{collectd}/bcache_blkid_{backbase}")
+    blkid_tpl = "blkid -o export {devpath} > {collectd}/blkid_{basename}"
+    block_size = textwrap.dedent("""
+        kname=$(basename `readlink {devpath}`)
+        q=/sys/class/block/$kname/queue
+        cat $q/logical_block_size > {collectd}/lbs_$kname
+        cat $q/physical_block_size > {collectd}/pbs_$kname
+        blockdev --getbsz /dev/$kname > {collectd}/blockdev_getbsz_$kname
+        blockdev --getpbsz /dev/$kname > {collectd}/blockdev_getpbsz_$kname
+        blockdev --getss /dev/$kname > {collectd}/blockdev_getss_$kname
+        blockdev --getsz /dev/$kname > {collectd}/blockdev_getsz_$kname""")
+    parted_tpl = ("parted --script {devpath} unit B print > "
+                  "{collectd}/parted_{basename}")
+    pvdisplay = ("pvdisplay -C --separator = -o vg_name,pv_name"
+                 " --noheadings > {collectd}/pvs")
+    lvdisplay = ("lvdisplay -C --separator = -o lv_name,vg_name"
+                 " --noheadings > {collectd}/lvs")
+    dmsetup = ("dmsetup info --noheading -c -o vg_name,lv_name,blkdevname"
+               " > {collectd}/dmsetup_info")
+    mdadm_status = "mdadm --detail --scan > {collectd}/mdadm_status"
+    mdadm_active = ("mdadm --detail --scan | grep -c ubuntu"
+                    " > {collectd}/mdadm_active")
+    proc_mdstat = "cat /proc/mdstat > {collectd}/proc_mdstat"
+    format_tpl = "{dumpfs} {devpath} > {collectd}/format_{basename}"
+
+    script_templates = {
+        'disk': [blkid_tpl, parted_tpl, block_size],
+        'partition': [blkid_tpl],
+        'lvm_volgroup': [pvdisplay, dmsetup],
+        'lvm_partition': [lvdisplay],
+        'raid': [mdadm_status, mdadm_active, proc_mdstat],
+        'bcache': [bcache_fs, bcache_cache_mode_map, bcache_backing_map,
+                   bcache_super_cache, bcache_super_back, bcache_blkid],
+        'format': [blkid_tpl, format_tpl],
+        'mount': [],
+    }
+
+    format_dumpfs = {
+        'fat32': '/bin/true',
+        'ext2': 'dumpe2fs -h',
+        'ext3': 'dumpe2fs -h',
+        'ext4': 'dumpe2fs -h',
+        'btrfs': 'btrfs-show-super',
+        'xfs': 'xfs_metadump',
+    }
+
+    storage_scripts = []
+    for item_id, command in storage_config.items():
+        templates = script_templates.get(command['type'], [])
+        devpath = get_path_to_storage_volume_no_sync(item_id,
+                                                     storage_config)
+        cachedev = storage_config.get(command.get('cache_device', ''))
+        cachebase = ''
+        if cachedev:
+            cachedev = get_path_to_storage_volume_no_sync(cachedev.get('id'),
+                                                          storage_config)
+            cachebase = os.path.basename(cachedev)
+
+        backdev = storage_config.get(command.get('backing_device', ''))
+        backbase = ''
+        if backdev:
+            backdev = get_path_to_storage_volume_no_sync(backdev.get('id'),
+                                                         storage_config)
+            backbase = os.path.basename(backdev)
+
+        basename = os.path.basename(devpath)
+        for template in templates:
+            data = {
+                'backbase': backbase,
+                'backdev': backdev,
+                'cachebase': cachebase,
+                'cachedev': cachedev,
+                'collectd': 'OUTPUT_COLLECT_D',
+                'devpath': devpath,
+                'basename': basename,
+                'dumpfs': format_dumpfs.get(command.get('fstype', ''), '')
+            }
+            print(template)
+            script = textwrap.dedent(template.format(**data).rstrip() + " |:")
+            logger.debug('Adding storage script: %s', script)
+            storage_scripts.append(script)
+
+    return storage_scripts
+
+
 class VMBaseClass(TestCase):
     __test__ = False
     arch_skip = []
@@ -355,11 +462,12 @@
     collect_scripts = []
     conf_file = "examples/tests/basic.yaml"
     disk_block_size = 512
-    disk_driver = 'virtio-blk'
+    disk_driver = 'scsi-hd'
     disk_to_check = {}
     extra_disks = []
     extra_kern_args = None
     fstab_expected = {}
+    generate_storage_scripts = True
     image_store_class = ImageStore
     install_timeout = 3000
     interactive = False
@@ -395,10 +503,55 @@
         logger.debug("Image %s\n  boot=%s\n  kernel=%s\n  initrd=%s\n"
                      "  tarball=%s\n", img_verstr, boot_img, boot_kernel,
                      boot_initrd, tarball)
+
+        try:
+            if cls.arch == 'ppc64el':
+                cls.conf_yaml = inject_prep_partition(cls.conf_file)
+            else:
+                cls.conf_yaml = yaml.load(util.load_file(cls.conf_file))
+            cls.storage_config_dict = (
+                extract_storage_ordered_dict(cls.conf_yaml))
+        except ValueError:
+            cls.storage_config_dict = {}
+            logger.warning('No storage config in %s, no storage scripts',
+                           cls.conf_file)
+            pass
+
+        # generate storage scripts as needed
+        storage_scripts = []
+        storage_scripts_modified = []
+        cls.storage_volume_dict = {}
+        if cls.generate_storage_scripts:
+            logger.info('Generating storage scripts from %s', cls.conf_file)
+            storage_scripts = generate_storage_scripts(cls.storage_config_dict)
+
+            # map storage config to volume paths
+            scd = cls.storage_config_dict
+            for item_id, command in cls.storage_config_dict.items():
+                devpath = get_path_to_storage_volume_no_sync(item_id, scd)
+                cls.storage_volume_dict.update({item_id: devpath})
+
+        # precise does not have virtio-scsi, so we use wwn to predict the
+        # path name, but know that it will be a virtio device with serial
+        if not cls.has_virtio_scsi() and 'storage' in cls.conf_yaml:
+            logger.info('updating script paths for precise workaround')
+            for script in storage_scripts:
+                script = script.replace("wwn-", "virtio-")
+                storage_scripts_modified.append(script)
+            logger.info('updating volume devpaths for precise workaround')
+            for item_id, devpath in cls.storage_volume_dict.items():
+                new_path = devpath.replace('wwn-', 'virtio-')
+                cls.storage_volume_dict.update({item_id: new_path})
+
+        # always use the class specific scripts
+        collect_scripts = cls.collect_scripts
+        # append any generated or modified scripts as well
+        collect_scripts += storage_scripts + storage_scripts_modified
+
         # set up tempdir
         cls.td = TempDir(
             name=cls.__name__,
-            user_data=generate_user_data(collect_scripts=cls.collect_scripts))
+            user_data=generate_user_data(collect_scripts=collect_scripts))
         logger.info('Using tempdir: %s , Image: %s', cls.td.tmpdir,
                     img_verstr)
         cls.install_log = os.path.join(cls.td.logs, 'install-serial.log')
@@ -451,18 +604,27 @@
 
         # build disk arguments
         disks = []
-        sc = util.load_file(cls.conf_file)
-        storage_config = yaml.load(sc).get('storage', {}).get('config', {})
-        cls.disk_wwns = ["wwn=%s" % x.get('wwn') for x in storage_config
+
+        # modify config for precise's lack of virtio-scsi
+        if not cls.has_virtio_scsi() and 'storage' in cls.conf_yaml:
+            logger.info('Precise detected, modifying storage config for test')
+            cls.conf_yaml = switch_wwn_for_serial_disks(cls.conf_yaml)
+            cls.storage_config_dict = (
+                extract_storage_ordered_dict(cls.conf_yaml))
+
+        cls.disk_wwns = ["wwn=%s" % x.get('wwn')
+                         for x in cls.storage_config_dict.values()
                          if 'wwn' in x]
         cls.disk_serials = ["serial=%s" % x.get('serial')
-                            for x in storage_config if 'serial' in x]
+                            for x in cls.storage_config_dict.values()
+                            if 'serial' in x]
 
         target_disk = "{}:{}:{}:{}:".format(cls.td.target_disk,
                                             "",
                                             cls.disk_driver,
                                             cls.disk_block_size)
-        if len(cls.disk_wwns):
+
+        if len(cls.disk_wwns) and cls.disk_driver == "scsi-hd":
             target_disk += cls.disk_wwns[0]
 
         if len(cls.disk_serials):
@@ -476,7 +638,7 @@
             extra_disk = '{}:{}:{}:{}:'.format(dpath, disk_sz,
                                                cls.disk_driver,
                                                cls.disk_block_size)
-            if len(cls.disk_wwns):
+            if len(cls.disk_wwns) and cls.disk_driver == "scsi-hd":
                 w_index = disk_no + 1
                 if w_index < len(cls.disk_wwns):
                     extra_disk += cls.disk_wwns[w_index]
@@ -496,8 +658,23 @@
                                                   "serial=nvme-%d" % disk_no)
             disks.extend(['--disk', nvme_disk])
 
+        configs = [cls.conf_file]
+        # write out modified config yaml if needed
+        if cls.arch.startswith('ppc64') or not cls.has_virtio_scsi():
+            if 'storage' in cls.conf_yaml:
+                conf_name = "modified-" + os.path.basename(cls.conf_file)
+                modified_conf_file = os.path.join(cls.td.install, conf_name)
+                logger.info('storage config modified for testing')
+                logger.debug('modified_config:\n%s', cls.conf_yaml)
+                with open(modified_conf_file, "w") as fp:
+                    fp.write(yaml.dump(cls.conf_yaml, default_flow_style=False,
+                                       indent=4))
+
+                shutil.copyfile(modified_conf_file,
+                                os.path.join(cls.td.logs, conf_name))
+                configs = [modified_conf_file]
+
         # proxy config
-        configs = [cls.conf_file]
         proxy = get_apt_proxy()
         if get_apt_proxy is not None:
             proxy_config = os.path.join(cls.td.install, 'proxy.cfg')
@@ -577,7 +754,7 @@
         for (disk_no, disk) in enumerate([cls.td.target_disk]):
             disk = '--disk={},driver={},format={},{}'.format(
                 disk, cls.disk_driver, TARGET_IMAGE_FORMAT, bsize_args)
-            if len(cls.disk_wwns):
+            if len(cls.disk_wwns) and cls.disk_driver == "scsi-hd":
                 disk += ",%s" % cls.disk_wwns[0]
             if len(cls.disk_serials):
                 disk += ",%s" % cls.disk_serials[0]
@@ -589,7 +766,7 @@
             dpath = os.path.join(cls.td.disks, 'extra_disk_%d.img' % disk_no)
             disk = '--disk={},driver={},format={},{}'.format(
                 dpath, cls.disk_driver, TARGET_IMAGE_FORMAT, bsize_args)
-            if len(cls.disk_wwns):
+            if len(cls.disk_wwns) and cls.disk_driver == "scsi-hd":
                 w_index = disk_no + 1
                 if w_index < len(cls.disk_wwns):
                     disk += ",%s" % cls.disk_wwns[w_index]
@@ -697,6 +874,14 @@
                           keep_fail=KEEP_DATA['fail'])
 
     @classmethod
+    def has_virtio_scsi(cls):
+        if cls.release == "precise" and cls.krel in [None, "precise"]:
+            cls.disk_driver = "virtio-blk"
+            return False
+
+        return True
+
+    @classmethod
     def expected_interfaces(cls):
         expected = []
         interfaces = {}
@@ -745,7 +930,56 @@
         return boot_log_wrap(cls.__name__, myboot, cmd, console_log, timeout,
                              purpose)
 
+    @classmethod
+    def human2bytes(self, size):
+        return util.human2bytes(size)
+
     # Misc functions that are useful for many tests
+    def _resolve_volpath(self, vol_path, ls_input):
+        part_kname = os.path.basename(vol_path)
+        ls_data = None
+        # if needed, look up storage path in ls_id file which maps
+        # volume paths (/dev/disk/*)  to knames (/dev/XXX)
+        if vol_path.startswith('/dev/disk/'):
+            with open(os.path.join(self.td.collect, ls_input)) as fp:
+                ls_data = fp.read()
+
+            self.assertIn(part_kname, ls_data)
+            # extract the mapping
+            # <vol_base> -> ../../<kernel name>
+            # wwn-0x2b873dea52e628f8-part1 -> ../../sda1
+            id_list = ls_data.split()
+            partidx = id_list.index(part_kname)
+            part_kname = id_list[partidx + 2].replace("../../", "")
+
+        return (part_kname, ls_data)
+
+    def _kname_to_fsuuid(self, kname, ls_input):
+
+        # prepend ../../ path
+        kname_base = "../../" + os.path.basename(kname)
+        ls_data = None
+        uuid = None
+        with open(os.path.join(self.td.collect, ls_input)) as fp:
+            ls_data = fp.read()
+
+            self.assertIn(kname_base, ls_data)
+            # extract the mapping
+            # <uuid> -> ../../<kernel basename>
+            id_list = ls_data.split()
+            partidx = id_list.index(kname_base)
+            uuid = id_list[partidx - 2]
+
+        return (uuid, ls_data)
+
+    def _test_proc_partitions(self, kname):
+        # with a kname, look at proc_partitions for the size
+        with open(os.path.join(self.td.collect, 'proc_partitions')) as fp:
+            proc_partitions = fp.read()
+
+        self.assertIn(kname, proc_partitions)
+        return proc_partitions
+
     def output_files_exist(self, files):
         for f in files:
             self.assertTrue(os.path.exists(os.path.join(self.td.collect, f)))
@@ -769,9 +1003,32 @@
             # Python 2.
             self.assertRegexpMatches(s, r)
 
+    def get_bcache_data(self, bcache_file):
+        """ parse bcache data into dict """
+        data = util.load_file(os.path.join(self.td.collect, bcache_file))
+        ret = {}
+        for line in data.splitlines():
+            if line == "":
+                continue
+            val = line.split()
+            ret[val[0]] = val[1]
+        return ret
+
+    def get_bcache_backing_map(self):
+        """ parse backing device to bcache name into dict. """
+        bcache_file = "bcache_backing_map"
+        data = util.load_file(os.path.join(self.td.collect, bcache_file))
+        ret = {}
+        for line in data.splitlines():
+            if line == "":
+                continue
+            val = line.split()
+            ret[val[1]] = val[0]
+        return ret
+
     def get_blkid_data(self, blkid_file):
-        with open(os.path.join(self.td.collect, blkid_file)) as fp:
-            data = fp.read()
+        """ parse blkid data into dict """
+        data = util.load_file(os.path.join(self.td.collect, blkid_file))
         ret = {}
         for line in data.splitlines():
             if line == "":
@@ -780,6 +1037,32 @@
             ret[val[0]] = val[1]
         return ret
 
+    def get_parted_data(self, parted_file):
+        """ parse parted data into dict """
+        data = util.load_file(os.path.join(self.td.collect, parted_file))
+        ret = {}
+        splitdata = data.split("\n\n")
+
+        for line in splitdata[0].splitlines():
+            if line == "":
+                continue
+            val = line.split(':')
+            ret[val[0].strip()] = val[1].strip()
+
+        partitions = []
+        if len(splitdata) > 1:
+            # populate fields, parted prints the column names
+            header = splitdata[1].splitlines()[0]
+            # transform strings into suitable keys
+            fields = header.lower().replace('file system',
+                                            'filesystem').split()
+            partitions = []
+            for partline in splitdata[1].splitlines()[1:]:
+                partitions.append(dict(zip(fields, partline.split())))
+
+        ret['partitions'] = partitions
+        return ret
+
     def test_fstab(self):
         if (os.path.exists(self.td.collect + "fstab") and
                 self.fstab_expected is not None):
@@ -814,6 +1097,257 @@
         self.assertNotIn("/etc/network/interfaces.d/eth0.cfg",
                          interfacesd.split("\n"))
 
+    def test_output_files_exist(self):
+        self.output_files_exist(
+            ["fstab", "ls_dname", "ls_uuid", "ls_id", "proc_partitions"])
+
+    def test_disks_ptable(self):
+        """ test_disks_ptable: Test disks with 'ptable' key present """
+
+        if not self.generate_storage_scripts:
+            reason = "This test uses only custom storage tests"
+            raise SkipTest(reason)
+
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'disk' and 'ptable' in item]:
+            vol = self.storage_config_dict.get(id)
+            vol_path = self.storage_volume_dict.get(id)
+
+            # Trusty blkid does not export some required data
+            if self.release not in ['trusty', 'precise']:
+                blkid_file = "blkid_%s" % os.path.basename(vol_path)
+                blkid_info = self.get_blkid_data(blkid_file)
+
+                # blkid reports 'DOS' for type msdos, so translate config
+                cfg_to_blkid = {
+                    'msdos': 'dos',
+                    'gpt': 'gpt',
+                }
+                self.assertEquals(cfg_to_blkid.get(vol.get('ptable')),
+                                  blkid_info["PTTYPE"])
+            else:
+                # for precise, trusty! use parted
+                parted_file = "parted_%s" % os.path.basename(vol_path)
+                parted_info = self.get_parted_data(parted_file)
+                self.assertEqual(vol.get('ptable'),
+                                 parted_info['Partition Table'])
+
+    def test_partitions(self):
+        """ test_partitions: for each partition check (kernel path, size)
+            1. exists
+            2. kpath
+            3. size
+
+        """
+        if not self.generate_storage_scripts:
+            reason = "This test uses only custom storage tests"
+            raise SkipTest(reason)
+
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'partition']:
+
+            # determine storage path
+            vol = self.storage_config_dict.get(id)
+            vol_path = self.storage_volume_dict.get(id)
+
+            # ensure vol_path exists
+            (vol_kname, ls_id) = self._resolve_volpath(vol_path,
+                                                       ls_input='ls_id')
+            print("vol=%s" % vol)
+            print("vol_path=%s" % vol_path)
+            print("vol_kname=%s" % vol_kname)
+
+            # check for kname in proc_partitions data, and return contents
+            proc_partitions = self._test_proc_partitions(vol_kname)
+            pp_list = proc_partitions.split()
+            # proc partition line format is:
+            #  major minor #blocks(in 1KB) kname(dev_short)
+            #  8     1     3145728         sda1
+            partidx = pp_list.index(vol_kname)
+
+            # blocks is one element before kname
+            print("pp_item: %s" % pp_list[partidx - 1])
+            part_size_bytes = int(pp_list[partidx - 1]) * 1024
+            print("part_size_bytes=%s" % part_size_bytes)
+            expected_size = int(self.human2bytes(vol.get('size')))
+            # extended partitions have a fixed size
+            if 'flag' in vol and vol['flag'] == "extended":
+                expected_size = 1024
+            print("expected_size=%s" % expected_size)
+            self.assertEqual(part_size_bytes, expected_size)
+
+    def test_formats(self):
+        """ test_formats: for each format check (device, uuid and fstype)
+            1. underlying device matches
+            2. if UUID set, validate
+            3. fstype matches
+
+            type: format
+            id: sda1_root
+            fstype: ext4
+            volume: sda1
+
+        """
+        if not self.generate_storage_scripts:
+            reason = "This test uses only custom storage tests"
+            raise SkipTest(reason)
+
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'format']:
+            format_cfg = self.storage_config_dict.get(id)
+            real_dev = self.storage_config_dict.get(format_cfg.get('volume'))
+            vol_path = self.storage_volume_dict.get(format_cfg.get('volume'))
+            vol_base = os.path.basename(vol_path)
+            print("cfg: %s" % format_cfg)
+            print("vol_path=%s" % vol_path)
+            print("vol_base=%s" % vol_base)
+
+            if real_dev['type'] in ['bcache']:
+                print('format backed by bcache')
+                bdev = real_dev.get('backing_device')
+                bdev_path = self.storage_volume_dict.get(bdev)
+                print('bdev_path=%s' % bdev_path)
+
+                # look up backing name to map to bcacheN
+                (bkname, ls_id) = self._resolve_volpath(bdev_path,
+                                                        ls_input='ls_id')
+                print("resolved bdev -> %s" % bkname)
+                bdev_bcache = self.get_bcache_backing_map()
+                bcache_kname = bdev_bcache[bkname]
+                print("mapped %s to bcache kname %s" % (bkname, bcache_kname))
+                vol_base = bcache_kname
+            else:
+                # ensure vol_path exists
+                (vol_kname, ls_id) = self._resolve_volpath(vol_path,
+                                                           ls_input='ls_id')
+            print("vol_base=%s" % vol_base)
+
+            # blkid reports different values vs. config file
+            cfg_to_blkid = {
+                'fat32': 'vfat',
+                'ext3': 'ext3',
+                'ext4': 'ext4',
+                'btrfs': 'btrfs',
+                'xfs': 'xfs',
+            }
+            # check fstype in blkid output
+            blkid_file = "blkid_%s" % vol_base
+            blkid_info = self.get_blkid_data(blkid_file)
+            print("blkid_file=%s" % blkid_file)
+            print("blkid_info: %s" % blkid_info)
+            self.assertEqual(cfg_to_blkid.get(
+                             format_cfg.get('fstype', 'NO-FSTYPE-SET')),
+                             blkid_info['TYPE'])
+            print("%s fstype ok" % format_cfg.get('id'))
+
+            if 'uuid' in format_cfg:
+                uuid = format_cfg.get('uuid', 'NO-UUID-SET')
+                self.assertEqual(uuid, blkid_info['UUID'])
+                print("%s uuid ok" % format_cfg.get('id'))
+
+    def test_mounts(self):
+        """ test_mounts: for each mount check (device and mount point)
+
+            type: mount
+            id: sda1_root_mount
+            path: /
+            device: sda1_root
+        """
+        if not self.generate_storage_scripts:
+            reason = "This test uses only custom storage tests"
+            raise SkipTest(reason)
+
+        with open(os.path.join(self.td.collect, "fstab"), "r") as fp:
+            fstab = fp.read()
+
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'mount']:
+            mount_cfg = self.storage_config_dict.get(id)
+            fs_cfg = self.storage_config_dict.get(mount_cfg.get('device'))
+            real_dev = self.storage_config_dict.get(fs_cfg.get('volume'))
+            print("mount_cfg: %s" % mount_cfg)
+            print("fs_cfg: %s" % fs_cfg)
+            print("real_dev: %s" % real_dev)
+
+            if real_dev['type'] in ['lvm_partition']:
+                vol_path = (
+                    self.storage_volume_dict.get(mount_cfg.get('device')))
+                vol_base = os.path.basename(vol_path)
+
+                # check DEVNAME on device
+                blkid_file = "blkid_%s" % vol_base
+                blkid_info = self.get_blkid_data(blkid_file)
+                # most releases support DEVNAME in blkid
+                if 'DEVNAME' in blkid_info:
+                    self.assertEqual(vol_path, blkid_info['DEVNAME'])
+                fstab_device = vol_path
+
+            elif real_dev['type'] in ['bcache']:
+                print('mount backed by bcache')
+                # translate backing dev kname to bcacheN
+                # "bcache_raid1" -> "/dev/md0"
+                bdev = real_dev.get('backing_device')
+                print('backing_device_id=%s' % bdev)
+                bdev_path = self.storage_volume_dict.get(bdev)
+                print('bdev_path=%s' % bdev_path)
+
+                # look up backing name to map to bcacheN
+                (bkname, ls_id) = self._resolve_volpath(bdev_path,
+                                                        ls_input='ls_id')
+                print("resolved bdev -> %s" % bkname)
+                bdev_bcache = self.get_bcache_backing_map()
+                bcache_kname = bdev_bcache[bkname]
+                print("mapped %s to bcache kname %s" % (bkname, bcache_kname))
+
+                # look up the by-uuid symlink value based on bcache kname
+                (uuid, ls_uuid) = self._kname_to_fsuuid(bcache_kname,
+                                                        ls_input='ls_uuid')
+                # check uuid on device
+                blkid_file = "blkid_%s" % bcache_kname
+                print("Using blkid_file: %s" % blkid_file)
+                blkid_info = self.get_blkid_data(blkid_file)
+                self.assertEqual(uuid, blkid_info['UUID'])
+                fstab_device = "UUID=%s" % uuid
+
+            else:
+                vol_path = (
+                    self.storage_volume_dict.get(mount_cfg.get('device')))
+                vol_base = os.path.basename(vol_path)
+
+                # ensure vol_path exists
+                (kname, ls_id) = self._resolve_volpath(vol_path,
+                                                       ls_input='ls_id')
+
+                # look up the by-uuid symlink value based on
+                (uuid, ls_uuid) = self._kname_to_fsuuid(kname,
+                                                        ls_input='ls_uuid')
+
+                # check uuid on device
+                blkid_file = "blkid_%s" % vol_base
+                blkid_info = self.get_blkid_data(blkid_file)
+                self.assertEqual(uuid, blkid_info['UUID'])
+                fstab_device = "UUID=%s" % uuid
+
+            # check fstab_device and path
+            print("checking fstab_device=%s" % fstab_device)
+            for line in fstab.split("\n"):
+                if fstab_device in line:
+                    (dev, mp, fs, op, _, _) = line.split()
+                    self.assertEqual(fstab_device, dev)
+                    self.assertEqual(mount_cfg.get('path', 'PATH-MISSING'), mp)
+                    print("%s mount ok" % mount_cfg.get('id'))
+
+    def test_proxy_set(self):
+        expected = get_apt_proxy()
+        with open(os.path.join(self.td.collect, "apt-proxy")) as fp:
+            apt_proxy_found = fp.read().rstrip()
+        if expected:
+            # the proxy should have gotten set through
+            self.assertIn(expected, apt_proxy_found)
+        else:
+            # no proxy, so the output of apt-config dump should be empty
+            self.assertEqual("", apt_proxy_found)
+
     def run(self, result):
         super(VMBaseClass, self).run(result)
         self.record_result(result)
@@ -919,6 +1453,24 @@
     def test_interfacesd_eth0_removed(self):
         pass
 
+    def test_output_files_exist(self):
+        pass
+
+    def test_disks_ptable(self):
+        pass
+
+    def test_partitions(self):
+        pass
+
+    def test_proxy_set(self):
+        pass
+
+    def test_formats(self):
+        pass
+
+    def test_mounts(self):
+        pass
+
     def _maybe_raise(self, exc):
         if self.allow_test_fails:
             raise exc
@@ -1018,8 +1570,27 @@
         shutdown -P now "Shutting down on precise"
         """)
 
-    scripts = ([collect_prep] + collect_scripts + [collect_post] +
-               [precise_poweroff])
+    # we always collect the following
+    default_collect = textwrap.dedent("""#!/bin/sh -x
+        cd OUTPUT_COLLECT_D
+        cat /proc/partitions > proc_partitions
+        cat /etc/fstab > fstab
+        mkdir -p /dev/disk/by-dname
+        ls /dev/disk/by-dname/ > ls_dname
+        ls -al /dev/disk/by-uuid/ > ls_uuid
+        ls -al /dev/disk/by-id/ > ls_id
+        ls /sys/firmware/efi/ > ls_sys_firmware_efi
+        find /etc/network/interfaces.d > find_interfacesd
+
+        v=""
+        out=$(apt-config shell v Acquire::HTTP::Proxy)
+        eval "$out"
+        echo "$v" > apt-proxy
+        """)
+
+    scripts = ([collect_prep] + [default_collect] + collect_scripts +
+               [collect_post] + [precise_poweroff])
+    logger.debug('Using %s collect scripts', len(scripts))
 
     for part in scripts:
         if not part.startswith("#!"):
@@ -1093,5 +1664,78 @@
     return ret
 
 
+def prep_partition_for_device(device):
+    """ Storage config for a Power PReP partition, both
+        msdos and gpt format"""
+    return {
+        'id': 'prep_partition',
+        'type': 'partition',
+        'size': '8M',
+        'flag': 'prep',
+        'guid': '9e1a2d38-c612-4316-aa26-8b49521e5a8b',
+        'offset': '1M',
+        'wipe': 'zero',
+        'grub_device': True,
+        'device': device}
+
+
+def inject_prep_partition(storage_conf_file):
+    """ Parse a curtin configuration file and inject
+        a prep partition if possible"""
+    conf_yaml = yaml.load(util.load_file(storage_conf_file))
+    storage_config = conf_yaml.get('storage', {}).get('config', {})
+    if len(storage_config) == 0:
+        logger.debug('No storage config, skipping prep injection')
+        return conf_yaml
+
+    all_disks = [item for item in storage_config
+                 if item['type'] == 'disk']
+    main_disk = all_disks[0]
+    main_partitions = [part for part in storage_config
+                       if part['type'] == 'partition' and
+                       part['device'] == main_disk['id']]
+
+    # for msdos disks
+    # must have a primary partition spot
+    if main_disk['ptable'] == 'msdos':
+        primary_parts = [part for part in main_partitions
+                         if part.get('flag', '') not in ['extended',
+                                                         'logical']]
+        if len(primary_parts) > 3:
+            logger.error("Can't find a primary partition for prep partition")
+            raise ValueError
+
+        prep_partition = prep_partition_for_device(main_disk['id'])
+        last_partition = primary_parts[-1]
+        partition_index = storage_config.index(last_partition)
+        storage_config.insert(partition_index + 1, prep_partition)
+        if 'grub_device' in main_disk:
+            del main_disk['grub_device']
+
+    # for gpt, find the boot_partition and replace it
+    elif main_disk['ptable'] == 'gpt':
+        prep_partition = prep_partition_for_device(main_disk['id'])
+        [boot_partition] = [part for part in main_partitions
+                            if part.get('flag', '') in ['bios_grub']]
+        partition_index = storage_config.index(boot_partition)
+        storage_config[partition_index] = prep_partition
+
+        if 'grub_device' in main_disk:
+            del main_disk['grub_device']
+
+    return conf_yaml
+
+
+def switch_wwn_for_serial_disks(conf_yaml):
+    if 'storage' in conf_yaml:
+        cfg = conf_yaml.get('storage').get('config')
+        for disk in [item for item in cfg
+                     if item['type'] == 'disk' and 'wwn' in item]:
+            disk['serial'] = disk['wwn']
+            del disk['wwn']
+
+    return conf_yaml
+
+
 apply_keep_settings()
 logger = _initialize_logging()

=== modified file 'tests/vmtests/test_basic.py'
--- tests/vmtests/test_basic.py	2016-06-23 14:26:39 +0000
+++ tests/vmtests/test_basic.py	2016-07-19 15:04:36 +0000
@@ -1,188 +1,25 @@
-from . import (
-    VMBaseClass,
-    get_apt_proxy)
+from . import VMBaseClass
 from .releases import base_vm_classes as relbase
 
-import os
-import re
-import textwrap
-
 
 class TestBasicAbs(VMBaseClass):
     interactive = False
     conf_file = "examples/tests/basic.yaml"
+    # FIXME: generate extra_disks, and nvme from parsing config yaml
     extra_disks = ['128G', '128G', '4G']
     nvme_disks = ['4G']
+    # FIXME: update disk_to_check method to extract label and
+    #        and partition number from config yaml
     disk_to_check = [('main_disk', 1), ('main_disk', 2)]
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        blkid -o export /dev/vda > blkid_output_vda
-        blkid -o export /dev/vda1 > blkid_output_vda1
-        blkid -o export /dev/vda2 > blkid_output_vda2
-        btrfs-show-super /dev/vdd > btrfs_show_super_vdd
-        cat /proc/partitions > proc_partitions
-        ls -al /dev/disk/by-uuid/ > ls_uuid
-        cat /etc/fstab > fstab
-        mkdir -p /dev/disk/by-dname
-        ls /dev/disk/by-dname/ > ls_dname
-        find /etc/network/interfaces.d > find_interfacesd
-
-        v=""
-        out=$(apt-config shell v Acquire::HTTP::Proxy)
-        eval "$out"
-        echo "$v" > apt-proxy
-        """)]
-
-    def test_output_files_exist(self):
-        self.output_files_exist(
-            ["blkid_output_vda", "blkid_output_vda1", "blkid_output_vda2",
-             "btrfs_show_super_vdd", "fstab", "ls_dname", "ls_uuid",
-             "proc_partitions"])
-
-    def test_ptable(self):
-        blkid_info = self.get_blkid_data("blkid_output_vda")
-        self.assertEquals(blkid_info["PTTYPE"], "dos")
-
-    def test_partition_numbers(self):
-        # vde should have partitions 1 and 10
-        disk = "vde"
-        proc_partitions_path = os.path.join(self.td.collect,
-                                            'proc_partitions')
-        self.assertTrue(os.path.exists(proc_partitions_path))
-        found = []
-        with open(proc_partitions_path, 'r') as fp:
-            for line in fp.readlines():
-                if disk in line:
-                    found.append(line.split()[3])
-        # /proc/partitions should have 3 lines with 'vde' in them.
-        expected = [disk + s for s in ["", "1", "10"]]
-        self.assertEqual(found, expected)
-
-    def test_partitions(self):
-        with open(os.path.join(self.td.collect, "fstab")) as fp:
-            fstab_lines = fp.readlines()
-        print("\n".join(fstab_lines))
-        # Test that vda1 is on /
-        blkid_info = self.get_blkid_data("blkid_output_vda1")
-        fstab_entry = None
-        for line in fstab_lines:
-            if blkid_info['UUID'] in line:
-                fstab_entry = line
-                break
-        self.assertIsNotNone(fstab_entry)
-        self.assertEqual(fstab_entry.split(' ')[1], "/")
-
-        # Test that vda2 is on /home
-        blkid_info = self.get_blkid_data("blkid_output_vda2")
-        fstab_entry = None
-        for line in fstab_lines:
-            if blkid_info['UUID'] in line:
-                fstab_entry = line
-                break
-        self.assertIsNotNone(fstab_entry)
-        self.assertEqual(fstab_entry.split(' ')[1], "/home")
-
-        # Test whole disk vdd is mounted at /btrfs
-        fstab_entry = None
-        for line in fstab_lines:
-            if "/dev/vdd" in line:
-                fstab_entry = line
-                break
-        self.assertIsNotNone(fstab_entry)
-        self.assertEqual(fstab_entry.split(' ')[1], "/btrfs")
-
-    def test_whole_disk_format(self):
-        # confirm the whole disk format is the expected device
-        with open(os.path.join(self.td.collect,
-                  "btrfs_show_super_vdd"), "r") as fp:
-            btrfs_show_super = fp.read()
-
-        with open(os.path.join(self.td.collect, "ls_uuid"), "r") as fp:
-            ls_uuid = fp.read()
-
-        # extract uuid from btrfs superblock
-        btrfs_fsid = [line for line in btrfs_show_super.split('\n')
-                      if line.startswith('fsid\t\t')]
-        self.assertEqual(len(btrfs_fsid), 1)
-        btrfs_uuid = btrfs_fsid[0].split()[1]
-        self.assertTrue(btrfs_uuid is not None)
-
-        # extract uuid from /dev/disk/by-uuid on /dev/vdd
-        # parsing ls -al output on /dev/disk/by-uuid:
-        # lrwxrwxrwx 1 root root   9 Dec  4 20:02
-        #  d591e9e9-825a-4f0a-b280-3bfaf470b83c -> ../../vdg
-        vdd_uuid = [line.split()[8] for line in ls_uuid.split('\n')
-                    if 'vdd' in line]
-        self.assertEqual(len(vdd_uuid), 1)
-        vdd_uuid = vdd_uuid.pop()
-        self.assertTrue(vdd_uuid is not None)
-
-        # compare them
-        self.assertEqual(vdd_uuid, btrfs_uuid)
-
-    def test_proxy_set(self):
-        expected = get_apt_proxy()
-        with open(os.path.join(self.td.collect, "apt-proxy")) as fp:
-            apt_proxy_found = fp.read().rstrip()
-        if expected:
-            # the proxy should have gotten set through
-            self.assertIn(expected, apt_proxy_found)
-        else:
-            # no proxy, so the output of apt-config dump should be empty
-            self.assertEqual("", apt_proxy_found)
 
 
 class PreciseTestBasic(relbase.precise, TestBasicAbs):
     __test__ = True
-
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        blkid -o export /dev/vda > blkid_output_vda
-        blkid -o export /dev/vda1 > blkid_output_vda1
-        blkid -o export /dev/vda2 > blkid_output_vda2
-        btrfs-show /dev/vdd > btrfs_show_super_vdd
-        cat /proc/partitions > proc_partitions
-        ls -al /dev/disk/by-uuid/ > ls_uuid
-        cat /etc/fstab > fstab
-        mkdir -p /dev/disk/by-dname
-        ls /dev/disk/by-dname/ > ls_dname
-        find /etc/network/interfaces.d > find_interfacesd
-
-        v=""
-        out=$(apt-config shell v Acquire::HTTP::Proxy)
-        eval "$out"
-        echo "$v" > apt-proxy
-        """)]
-
-    def test_whole_disk_format(self):
-        # confirm the whole disk format is the expected device
-        with open(os.path.join(self.td.collect,
-                  "btrfs_show_super_vdd"), "r") as fp:
-            btrfs_show_super = fp.read()
-
-        with open(os.path.join(self.td.collect, "ls_uuid"), "r") as fp:
-            ls_uuid = fp.read()
-
-        # extract uuid from btrfs superblock
-        btrfs_fsid = re.findall('.*uuid:\ (.*)\n', btrfs_show_super)
-
-        self.assertEqual(len(btrfs_fsid), 1)
-        btrfs_uuid = btrfs_fsid.pop()
-        self.assertTrue(btrfs_uuid is not None)
-
-        # extract uuid from /dev/disk/by-uuid on /dev/vdd
-        # parsing ls -al output on /dev/disk/by-uuid:
-        # lrwxrwxrwx 1 root root   9 Dec  4 20:02
-        #  d591e9e9-825a-4f0a-b280-3bfaf470b83c -> ../../vdg
-        vdd_uuid = [line.split()[8] for line in ls_uuid.split('\n')
-                    if 'vdd' in line]
-        self.assertEqual(len(vdd_uuid), 1)
-        vdd_uuid = vdd_uuid.pop()
-        self.assertTrue(vdd_uuid is not None)
-
-        # compare them
-        self.assertEqual(vdd_uuid, btrfs_uuid)
-
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
+
+    # FIXME(LP: #1523037): dname does not work on precise, so we cannot expect
+    # sda-part2 to exist in /dev/disk/by-dname as we can on other releases
+    # when dname works on trusty, then we need to re-enable by removing line.
     def test_ptable(self):
         print("test_ptable does not work for Precise")
 
@@ -206,6 +43,7 @@
 class PreciseHWETTestBasic(relbase.precise_hwe_t, PreciseTestBasic):
     # FIXME: off due to test_whole_disk_format failing
     __test__ = False
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
 
 
 class TrustyHWEUTestBasic(relbase.trusty_hwe_u, TrustyTestBasic):
@@ -238,114 +76,6 @@
 class TestBasicScsiAbs(TestBasicAbs):
     conf_file = "examples/tests/basic_scsi.yaml"
     disk_driver = 'scsi-hd'
-    extra_disks = ['128G', '128G', '4G']
-    nvme_disks = ['4G']
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        blkid -o export /dev/sda > blkid_output_sda
-        blkid -o export /dev/sda1 > blkid_output_sda1
-        blkid -o export /dev/sda2 > blkid_output_sda2
-        btrfs-show-super /dev/sdc > btrfs_show_super_sdc
-        cat /proc/partitions > proc_partitions
-        ls -al /dev/disk/by-uuid/ > ls_uuid
-        ls -al /dev/disk/by-id/ > ls_disk_id
-        cat /etc/fstab > fstab
-        mkdir -p /dev/disk/by-dname
-        ls /dev/disk/by-dname/ > ls_dname
-        find /etc/network/interfaces.d > find_interfacesd
-
-        v=""
-        out=$(apt-config shell v Acquire::HTTP::Proxy)
-        eval "$out"
-        echo "$v" > apt-proxy
-        """)]
-
-    def test_output_files_exist(self):
-        self.output_files_exist(
-            ["blkid_output_sda", "blkid_output_sda1", "blkid_output_sda2",
-             "btrfs_show_super_sdc", "fstab", "ls_dname", "ls_uuid",
-             "ls_disk_id", "proc_partitions"])
-
-    def test_ptable(self):
-        blkid_info = self.get_blkid_data("blkid_output_sda")
-        self.assertEquals(blkid_info["PTTYPE"], "dos")
-
-    def test_partition_numbers(self):
-        # vde should have partitions 1 and 10
-        disk = "sdd"
-        proc_partitions_path = os.path.join(self.td.collect,
-                                            'proc_partitions')
-        self.assertTrue(os.path.exists(proc_partitions_path))
-        found = []
-        with open(proc_partitions_path, 'r') as fp:
-            for line in fp.readlines():
-                if disk in line:
-                    found.append(line.split()[3])
-        # /proc/partitions should have 3 lines with 'vde' in them.
-        expected = [disk + s for s in ["", "1", "10"]]
-        self.assertEqual(found, expected)
-
-    def test_partitions(self):
-        with open(os.path.join(self.td.collect, "fstab")) as fp:
-            fstab_lines = fp.readlines()
-        print("\n".join(fstab_lines))
-        # Test that vda1 is on /
-        blkid_info = self.get_blkid_data("blkid_output_sda1")
-        fstab_entry = None
-        for line in fstab_lines:
-            if blkid_info['UUID'] in line:
-                fstab_entry = line
-                break
-        self.assertIsNotNone(fstab_entry)
-        self.assertEqual(fstab_entry.split(' ')[1], "/")
-
-        # Test that vda2 is on /home
-        blkid_info = self.get_blkid_data("blkid_output_sda2")
-        fstab_entry = None
-        for line in fstab_lines:
-            if blkid_info['UUID'] in line:
-                fstab_entry = line
-                break
-        self.assertIsNotNone(fstab_entry)
-        self.assertEqual(fstab_entry.split(' ')[1], "/home")
-
-        # Test whole disk sdc is mounted at /btrfs
-        fstab_entry = None
-        for line in fstab_lines:
-            if "/dev/sdc" in line:
-                fstab_entry = line
-                break
-        self.assertIsNotNone(fstab_entry)
-        self.assertEqual(fstab_entry.split(' ')[1], "/btrfs")
-
-    def test_whole_disk_format(self):
-        # confirm the whole disk format is the expected device
-        with open(os.path.join(self.td.collect,
-                  "btrfs_show_super_sdc"), "r") as fp:
-            btrfs_show_super = fp.read()
-
-        with open(os.path.join(self.td.collect, "ls_uuid"), "r") as fp:
-            ls_uuid = fp.read()
-
-        # extract uuid from btrfs superblock
-        btrfs_fsid = [line for line in btrfs_show_super.split('\n')
-                      if line.startswith('fsid\t\t')]
-        self.assertEqual(len(btrfs_fsid), 1)
-        btrfs_uuid = btrfs_fsid[0].split()[1]
-        self.assertTrue(btrfs_uuid is not None)
-
-        # extract uuid from /dev/disk/by-uuid on /dev/sdc
-        # parsing ls -al output on /dev/disk/by-uuid:
-        # lrwxrwxrwx 1 root root   9 Dec  4 20:02
-        #  d591e9e9-825a-4f0a-b280-3bfaf470b83c -> ../../vdg
-        uuid = [line.split()[8] for line in ls_uuid.split('\n')
-                if 'sdc' in line]
-        self.assertEqual(len(uuid), 1)
-        uuid = uuid.pop()
-        self.assertTrue(uuid is not None)
-
-        # compare them
-        self.assertEqual(uuid, btrfs_uuid)
 
 
 class XenialTestScsiBasic(relbase.xenial, TestBasicScsiAbs):

=== modified file 'tests/vmtests/test_bcache_basic.py'
--- tests/vmtests/test_bcache_basic.py	2016-06-13 20:49:15 +0000
+++ tests/vmtests/test_bcache_basic.py	2016-07-19 15:04:36 +0000
@@ -1,43 +1,160 @@
 from . import VMBaseClass
 from .releases import base_vm_classes as relbase
 
-import textwrap
 import os
 
 
 class TestBcacheBasic(VMBaseClass):
     arch_skip = [
         "s390x",  # lp:1565029
+        "ppc64el",  # LP:1602299
     ]
     conf_file = "examples/tests/bcache_basic.yaml"
     extra_disks = ['2G']
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        bcache-super-show /dev/vda2 > bcache_super_vda2
-        ls /sys/fs/bcache > bcache_ls
-        cat /sys/block/bcache0/bcache/cache_mode > bcache_cache_mode
-        cat /proc/mounts > proc_mounts
-        cat /proc/partitions > proc_partitions
-        find /etc/network/interfaces.d > find_interfacesd
-        """)]
+
+    def output_files_exist_bcache_basic(self):
+        to_check = ["ls_bcache"]
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'bcache']:
+            vol_base = os.path.basename(self.storage_volume_dict.get(id))
+            to_check.append("%s_cache_mode" % vol_base)
+            to_check.append("bcache_super_%s" % vol_base)
+
+        self.output_files_exist(to_check)
 
     def test_bcache_output_files_exist(self):
-        self.output_files_exist(["bcache_super_vda2", "bcache_ls",
-                                 "bcache_cache_mode"])
+        always_present = [
+            "bcache_backing_map",
+            "bcache_cache_mode_map",
+            "ls_bcache",
+        ]
+        per_device = []
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'bcache']:
+            bcache_cfg = self.storage_config_dict.get(id)
+
+            # backingdev related files
+            bdev = bcache_cfg.get('backing_device')
+            bdev_path = self.storage_volume_dict.get(bdev)
+            # look up backing name to map to bcacheN
+            (bkname, ls_id) = self._resolve_volpath(bdev_path,
+                                                    ls_input='ls_id')
+            bdev_bcache = self.get_bcache_backing_map()
+            bcache_kname = bdev_bcache[bkname]
+
+            blkid_file = "bcache_blkid_%s" % os.path.basename(bdev_path)
+            if blkid_file not in per_device:
+                per_device.append(blkid_file)
+
+            blkid_file = ("bcache_super_backing_%s" %
+                          os.path.basename(bdev_path))
+            if blkid_file not in per_device:
+                per_device.append(blkid_file)
+
+            blkid_file = "blkid_%s" % bcache_kname
+            if blkid_file not in per_device:
+                per_device.append(blkid_file)
+
+            # caching_device
+            cdev = bcache_cfg.get('cache_device')
+            cdev_path = self.storage_volume_dict.get(cdev)  # /dev/disk/by-id
+            blkid_file = "bcache_super_cache_%s" % os.path.basename(cdev_path)
+            if blkid_file not in per_device:
+                per_device.append(blkid_file)
+
+        to_check = always_present + per_device
+        print(to_check)
+        self.output_files_exist(to_check)
+
+    def bcache_status_basic(self):
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'bcache']:
+            vol_path = self.storage_volume_dict.get(id)
+            vol_base = os.path.basename(vol_path)
+
+            # check cset.uuid on cache devices matches sysfs/fs/bcache
+            bcache_super = "bcache_super_%s" % vol_base
+            bcache_super_info = self.get_bcache_data(bcache_super)
+            self.check_file_regex("ls_bcache", bcache_super_info['cset.uuid'])
 
     def test_bcache_status(self):
-        bcache_cset_uuid = None
-        fname = os.path.join(self.td.collect, "bcache_super_vda2")
-        with open(fname, "r") as fp:
-            for line in fp.read().splitlines():
-                if line != "" and line.split()[0] == "cset.uuid":
-                    bcache_cset_uuid = line.split()[-1].rstrip()
-        self.assertIsNotNone(bcache_cset_uuid)
-        with open(os.path.join(self.td.collect, "bcache_ls"), "r") as fp:
-            self.assertTrue(bcache_cset_uuid in fp.read().splitlines())
+        """test_bcache_status: match dev.uuid from superblock """
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'bcache']:
+            bcache_cfg = self.storage_config_dict.get(id)
+            print("bcache cfg: %s" % bcache_cfg)
+
+            # backing_device related names
+            bdev = bcache_cfg.get('backing_device')
+            print("backing_dev: %s" % bdev)
+            bdev_path = self.storage_volume_dict.get(bdev)
+            print("bdev_path: %s" % bdev_path)
+            # look up backing name to map to bcacheN
+            (bkname, ls_id) = self._resolve_volpath(bdev_path,
+                                                    ls_input='ls_id')
+            print("bkname: %s" % bkname)
+
+            (bk_uuid, ls_uuid) = self._kname_to_fsuuid(bkname,
+                                                       ls_input='ls_uuid')
+
+            bcache_super = ("bcache_super_backing_%s" %
+                            os.path.basename(bdev_path))
+            super_data = self.get_bcache_data(bcache_super)
+            dev_uuid = super_data['dev.uuid']
+            self.assertEqual(bk_uuid, dev_uuid)
+            print("backing_device uuid OK")
+
+            # cache_device related name
+            print("check cache dev")
+            cdev = bcache_cfg.get('cache_device')
+            print("cdev: %s" % cdev)
+            cdev_path = self.storage_volume_dict.get(cdev)  # /dev/disk/by-id
+            print("cdev_path: %s" % cdev_path)
+            # /dev/disk/byid -> kname
+            (ckname, ls_id) = self._resolve_volpath(cdev_path,
+                                                    ls_input='ls_id')
+            print("ckname: %s" % ckname)
+            (cc_uuid, ls_uuid) = self._kname_to_fsuuid(ckname,
+                                                       ls_input='ls_uuid')
+
+            bcache_super = ("bcache_super_cache_%s" %
+                            os.path.basename(cdev_path))
+            print("super_file: %s" % bcache_super)
+            super_data = self.get_bcache_data(bcache_super)
+            dev_uuid = super_data['dev.uuid']
+            print("super.dev.uuid: %s" % dev_uuid)
+            print("cdev.uuid: %s" % cc_uuid)
+            self.assertEqual(cc_uuid, dev_uuid)
+
+    def bcache_cachemode_basic(self):
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'bcache']:
+            vol = self.storage_config_dict.get(id)
+            vol_path = self.storage_volume_dict.get(id)
+            vol_base = os.path.basename(vol_path)
+            regex = r"\[%s]" % vol.get('cache_mode')
+            self.check_file_regex("%s_cache_mode" % vol_base, regex)
 
     def test_bcache_cachemode(self):
-        self.check_file_regex("bcache_cache_mode", r"\[writeback\]")
+        """test_bcache_cachemode: validate bcache cache setting"""
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'bcache']:
+            bcache_cfg = self.storage_config_dict.get(id)
+            bdev = bcache_cfg.get('backing_device')
+            bdev_path = self.storage_volume_dict.get(bdev)
+            print('bdev_path=%s' % bdev_path)
+
+            # look up backing name to map to bcacheN
+            (bkname, ls_id) = self._resolve_volpath(bdev_path,
+                                                    ls_input='ls_id')
+            print("resolved bdev -> %s" % bkname)
+            bdev_bcache = self.get_bcache_backing_map()
+
+            bcache_kname = bdev_bcache[bkname]
+            print("mapped %s to bcache kname %s" % (bkname, bcache_kname))
+
+            regex = r"%s:.*[%s]" % (bcache_kname, bcache_cfg.get('cache_mode'))
+            self.check_file_regex("bcache_cache_mode_map", regex)
 
 
 class PreciseHWETBcacheBasic(relbase.precise_hwe_t, TestBcacheBasic):

=== modified file 'tests/vmtests/test_bonding.py'
--- tests/vmtests/test_bonding.py	2016-06-15 19:38:44 +0000
+++ tests/vmtests/test_bonding.py	2016-07-19 15:04:36 +0000
@@ -167,6 +167,7 @@
 
 class PreciseHWETTestBonding(relbase.precise_hwe_t, TestNetworkAbs):
     __test__ = True
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
     # package names on precise are different, need to check on ifenslave-2.6
     collect_scripts = TestNetworkAbs.collect_scripts + [textwrap.dedent("""
              cd OUTPUT_COLLECT_D

=== modified file 'tests/vmtests/test_lvm.py'
--- tests/vmtests/test_lvm.py	2016-06-13 20:49:15 +0000
+++ tests/vmtests/test_lvm.py	2016-07-19 15:04:36 +0000
@@ -1,21 +1,11 @@
 from . import VMBaseClass
 from .releases import base_vm_classes as relbase
 
-import textwrap
-
 
 class TestLvmAbs(VMBaseClass):
     conf_file = "examples/tests/lvm.yaml"
     interactive = False
     extra_disks = []
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        cat /etc/fstab > fstab
-        ls /dev/disk/by-dname > ls_dname
-        find /etc/network/interfaces.d > find_interfacesd
-        pvdisplay -C --separator = -o vg_name,pv_name --noheadings > pvs
-        lvdisplay -C --separator = -o lv_name,vg_name --noheadings > lvs
-        """)]
     fstab_expected = {
         '/dev/vg1/lv1': '/srv/data',
         '/dev/vg1/lv2': '/srv/backup',
@@ -26,21 +16,58 @@
                      ('vg1-lv1', 0),
                      ('vg1-lv2', 0)]
 
-    def test_lvs(self):
-        self.check_file_strippedline("lvs", "lv1=vg1")
-        self.check_file_strippedline("lvs", "lv2=vg1")
-
-    def test_pvs(self):
-        self.check_file_strippedline("pvs", "vg1=/dev/vda5")
-        self.check_file_strippedline("pvs", "vg1=/dev/vda6")
-
-    def test_output_files_exist(self):
-        self.output_files_exist(
-            ["fstab", "ls_dname"])
+    def test_lvm_volgroup(self):
+        """ test_lvm_volgroups: for each lvm_volgroup check (devices, name)
+            1. underlying devices are members of the vg
+            2. validate the vg name is present
+
+        """
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'lvm_volgroup']:
+
+            # determine storage path
+            vol = self.storage_config_dict.get(id)
+            vol_path = self.storage_volume_dict.get(id)
+
+            # ensure vol_path exists
+            (vol_kname, ls_id) = self._resolve_volpath(vol_path,
+                                                       ls_input='ls_id')
+
+            for device in vol.get('devices'):
+                # lookup each device member and resolve path
+                dev_vol_path = self.storage_volume_dict.get(device)
+                (devpath, _) = self._resolve_volpath(dev_vol_path,
+                                                     ls_input='ls_id')
+                pvs_str = "%s=%s" % (vol.get('name'), "/dev/"+devpath)
+                self.check_file_strippedline("pvs", pvs_str)
+
+    def test_lvm_partition(self):
+        """ test_lvm_partitions: for each lvm_partitions check (devices, name)
+            1. device exists
+            2. validate the lv name is present in the lvs listing
+
+        """
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'lvm_partition']:
+
+            # determine storage path
+            vol = self.storage_config_dict.get(id)
+
+            # extract the volgroup name the lvm_partition specified
+            lv_name = vol.get('name')
+            vg_id = vol.get('volgroup')
+            vg_name = self.storage_config_dict.get(vg_id).get('name')
+
+            lvs_str = "%s=%s" % (lv_name, vg_name)
+            self.check_file_strippedline("lvs", lvs_str)
+
+    def test_output_files_exist_lvm(self):
+        self.output_files_exist(["pvs", "lvs", "dmsetup_info"])
 
 
 class PreciseTestLvm(relbase.precise, TestLvmAbs):
     __test__ = True
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
 
     # FIXME(LP: #1523037): dname does not work on trusty, so we cannot expect
     # sda-part2 to exist in /dev/disk/by-dname as we can on other releases

=== modified file 'tests/vmtests/test_mdadm_bcache.py'
--- tests/vmtests/test_mdadm_bcache.py	2016-06-13 20:49:15 +0000
+++ tests/vmtests/test_mdadm_bcache.py	2016-07-19 15:04:36 +0000
@@ -1,39 +1,31 @@
 from . import VMBaseClass
+from .test_bcache_basic import TestBcacheBasic
 from .releases import base_vm_classes as relbase
 
 import textwrap
-import os
 
 
 class TestMdadmAbs(VMBaseClass):
     interactive = False
     extra_disks = []
     active_mdadm = "1"
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        cat /etc/fstab > fstab
-        mdadm --detail --scan > mdadm_status
-        mdadm --detail --scan | grep -c ubuntu > mdadm_active1
-        grep -c active /proc/mdstat > mdadm_active2
-        ls /dev/disk/by-dname > ls_dname
-        find /etc/network/interfaces.d > find_interfacesd
-        """)]
 
     def test_mdadm_output_files_exist(self):
         self.output_files_exist(
-            ["fstab", "mdadm_status", "mdadm_active1", "mdadm_active2",
+            ["fstab", "mdadm_status", "mdadm_active", "proc_mdstat",
              "ls_dname"])
 
     def test_mdadm_status(self):
         # ubuntu:<ID> is the name assigned to the md array
         self.check_file_regex("mdadm_status", r"ubuntu:[0-9]*")
-        self.check_file_strippedline("mdadm_active1", self.active_mdadm)
-        self.check_file_strippedline("mdadm_active2", self.active_mdadm)
-
-
-class TestMdadmBcacheAbs(TestMdadmAbs):
+        self.check_file_strippedline("mdadm_active", self.active_mdadm)
+        self.check_file_regex("proc_mdstat", r"active")
+
+
+class TestMdadmBcacheAbs(TestBcacheBasic, TestMdadmAbs):
     arch_skip = [
         "s390x",  # lp:1565029
+        "ppc64el",  # LP:1602299
         ]
     conf_file = "examples/tests/mdadm_bcache.yaml"
     disk_to_check = [('main_disk', 1),
@@ -46,76 +38,10 @@
                      ('cached_array', 0),
                      ('cached_array_2', 0)]
     extra_disks = ['4G', '4G']
-    collect_scripts = TestMdadmAbs.collect_scripts + [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        bcache-super-show /dev/vda6 > bcache_super_vda6
-        bcache-super-show /dev/vda7 > bcache_super_vda7
-        bcache-super-show /dev/md0 > bcache_super_md0
-        ls /sys/fs/bcache > bcache_ls
-        cat /sys/block/bcache0/bcache/cache_mode > bcache_cache_mode
-        cat /sys/block/bcache1/bcache/cache_mode >> bcache_cache_mode
-        cat /sys/block/bcache2/bcache/cache_mode >> bcache_cache_mode
-        cat /proc/mounts > proc_mounts
-        find /etc/network/interfaces.d > find_interfacesd
-        """)]
-    fstab_expected = {
-        '/dev/vda1': '/media/sda1',
-        '/dev/vda7': '/boot',
-        '/dev/bcache1': '/media/data',
-        '/dev/bcache0': '/media/bcache_normal',
-        '/dev/bcache2': '/media/bcachefoo_fulldiskascache_storage'
-    }
-
-    def test_bcache_output_files_exist(self):
-        self.output_files_exist(["bcache_super_vda6",
-                                 "bcache_super_vda7",
-                                 "bcache_super_md0",
-                                 "bcache_ls",
-                                 "bcache_cache_mode"])
-
-    def test_bcache_status(self):
-        bcache_supers = [
-            "bcache_super_vda6",
-            "bcache_super_vda7",
-            "bcache_super_md0",
-        ]
-        bcache_cset_uuid = None
-        found = {}
-        for bcache_super in bcache_supers:
-            with open(os.path.join(self.td.collect, bcache_super), "r") as fp:
-                for line in fp.read().splitlines():
-                    if line != "" and line.split()[0] == "cset.uuid":
-                        bcache_cset_uuid = line.split()[-1].rstrip()
-                        if bcache_cset_uuid in found:
-                            found[bcache_cset_uuid].append(bcache_super)
-                        else:
-                            found[bcache_cset_uuid] = [bcache_super]
-            self.assertIsNotNone(bcache_cset_uuid)
-            with open(os.path.join(self.td.collect, "bcache_ls"), "r") as fp:
-                self.assertTrue(bcache_cset_uuid in fp.read().splitlines())
-
-        # one cset.uuid for all devices
-        self.assertEqual(len(found), 1)
-
-        # three devices with same cset.uuid
-        self.assertEqual(len(found[bcache_cset_uuid]), 3)
-
-        # check the cset.uuid in the dict
-        self.assertEqual(list(found.keys()).pop(),
-                         bcache_cset_uuid)
-
-    def test_bcache_cachemode(self):
-        # definition is on order 0->back,1->through,2->around
-        # but after reboot it can be anything since order is not guaranteed
-        # until we find a way to redetect the order we just check that all
-        # three are there
-        self.check_file_regex("bcache_cache_mode", r"\[writeback\]")
-        self.check_file_regex("bcache_cache_mode", r"\[writethrough\]")
-        self.check_file_regex("bcache_cache_mode", r"\[writearound\]")
 
 
 class TrustyTestMdadmBcache(relbase.trusty, TestMdadmBcacheAbs):
-    __test__ = True
+    __test__ = False  # Stack traces for now
 
     # FIXME(LP: #1523037): dname does not work on trusty
     # when dname works on trusty, then we need to re-enable by removing line.

=== modified file 'tests/vmtests/test_multipath.py'
--- tests/vmtests/test_multipath.py	2016-06-23 14:26:39 +0000
+++ tests/vmtests/test_multipath.py	2016-07-19 15:04:36 +0000
@@ -11,6 +11,7 @@
     disk_driver = 'scsi-hd'
     extra_disks = []
     nvme_disks = []
+    generate_storage_scripts = False
     collect_scripts = [textwrap.dedent("""
         cd OUTPUT_COLLECT_D
         blkid -o export /dev/sda > blkid_output_sda

=== modified file 'tests/vmtests/test_network.py'
--- tests/vmtests/test_network.py	2016-06-15 19:38:44 +0000
+++ tests/vmtests/test_network.py	2016-07-19 15:04:36 +0000
@@ -322,12 +322,14 @@
 
 
 class PreciseHWETTestNetwork(relbase.precise_hwe_t, TestNetworkAbs):
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
     # FIXME: off due to hang at test: Starting execute cloud user/final scripts
     __test__ = False
 
 
 class PreciseHWETTestNetworkStatic(relbase.precise_hwe_t,
                                    TestNetworkStaticAbs):
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
     # FIXME: off due to hang at test: Starting execute cloud user/final scripts
     __test__ = False
 
@@ -398,6 +400,7 @@
 
 
 class PreciseTestNetworkVlan(relbase.precise, TestNetworkVlanAbs):
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
     __test__ = True
 
     # precise ip -d link show output is different (of course)
@@ -430,6 +433,7 @@
 
 
 class PreciseTestNetworkENISource(relbase.precise, TestNetworkENISource):
+    arch_skip = ["ppc64el", "ppc64le", "s390x"]
     __test__ = False
     # not working, still debugging though; possible older ifupdown doesn't
     # like the multiple iface method.

=== modified file 'tests/vmtests/test_nvme.py'
--- tests/vmtests/test_nvme.py	2016-06-13 20:49:15 +0000
+++ tests/vmtests/test_nvme.py	2016-07-19 15:04:36 +0000
@@ -9,15 +9,17 @@
     arch_skip = [
         "s390x",  # nvme is a pci device, no pci on s390x
     ]
+    generate_storage_scripts = False
     interactive = False
     conf_file = "examples/tests/nvme.yaml"
     install_timeout = 600
     boot_timeout = 120
     extra_disks = []
     nvme_disks = ['4G', '4G']
-    disk_to_check = [('main_disk', 1), ('main_disk', 2), ('main_disk', 15),
+    disk_to_check = [('main_disk', 1), ('main_disk', 2),
                      ('nvme_disk', 1), ('nvme_disk', 2), ('nvme_disk', 3),
                      ('second_nvme', 1)]
+
     collect_scripts = [textwrap.dedent("""
         cd OUTPUT_COLLECT_D
         ls /sys/class/ > sys_class

=== modified file 'tests/vmtests/test_raid5_bcache.py'
--- tests/vmtests/test_raid5_bcache.py	2016-06-13 20:49:15 +0000
+++ tests/vmtests/test_raid5_bcache.py	2016-07-19 15:04:36 +0000
@@ -1,72 +1,14 @@
-from . import VMBaseClass
+from .test_bcache_basic import TestBcacheBasic
+from .test_mdadm_bcache import TestMdadmAbs
 from .releases import base_vm_classes as relbase
 
-import textwrap
-import os
-
-
-class TestMdadmAbs(VMBaseClass):
+
+class TestMdadmBcacheAbs(TestBcacheBasic, TestMdadmAbs):
     interactive = False
     extra_disks = ['10G', '10G', '10G', '10G']
-    active_mdadm = "1"
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        cat /etc/fstab > fstab
-        mdadm --detail --scan > mdadm_status
-        mdadm --detail --scan | grep -c ubuntu > mdadm_active1
-        grep -c active /proc/mdstat > mdadm_active2
-        ls /dev/disk/by-dname > ls_dname
-        find /etc/network/interfaces.d > find_interfacesd
-        """)]
-
-    def test_mdadm_output_files_exist(self):
-        self.output_files_exist(
-            ["fstab", "mdadm_status", "mdadm_active1", "mdadm_active2",
-             "ls_dname"])
-
-    def test_mdadm_status(self):
-        # ubuntu:<ID> is the name assigned to the md array
-        self.check_file_regex("mdadm_status", r"ubuntu:[0-9]*")
-        self.check_file_strippedline("mdadm_active1", self.active_mdadm)
-        self.check_file_strippedline("mdadm_active2", self.active_mdadm)
-
-
-class TestMdadmBcacheAbs(TestMdadmAbs):
     conf_file = "examples/tests/raid5bcache.yaml"
     disk_to_check = [('md0', 0), ('sda', 2)]
 
-    collect_scripts = TestMdadmAbs.collect_scripts + [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        bcache-super-show /dev/vda2 > bcache_super_vda2
-        ls /sys/fs/bcache > bcache_ls
-        cat /sys/block/bcache0/bcache/cache_mode > bcache_cache_mode
-        cat /proc/mounts > proc_mounts
-        cat /proc/partitions > proc_partitions
-        find /etc/network/interfaces.d > find_interfacesd
-        """)]
-    fstab_expected = {
-        '/dev/bcache0': '/',
-        '/dev/md0': '/srv/data',
-    }
-
-    def test_bcache_output_files_exist(self):
-        self.output_files_exist(["bcache_super_vda2", "bcache_ls",
-                                 "bcache_cache_mode"])
-
-    def test_bcache_status(self):
-        bcache_cset_uuid = None
-        fname = os.path.join(self.td.collect, "bcache_super_vda2")
-        with open(fname, "r") as fp:
-            for line in fp.read().splitlines():
-                if line != "" and line.split()[0] == "cset.uuid":
-                    bcache_cset_uuid = line.split()[-1].rstrip()
-        self.assertIsNotNone(bcache_cset_uuid)
-        with open(os.path.join(self.td.collect, "bcache_ls"), "r") as fp:
-            self.assertTrue(bcache_cset_uuid in fp.read().splitlines())
-
-    def test_bcache_cachemode(self):
-        self.check_file_regex("bcache_cache_mode", r"\[writeback\]")
-
 
 class PreciseHWETTestRaid5Bcache(relbase.precise_hwe_t, TestMdadmBcacheAbs):
     # FIXME: off due to failing install: RUN_ARRAY failed: Invalid argument

=== modified file 'tests/vmtests/test_uefi_basic.py'
--- tests/vmtests/test_uefi_basic.py	2016-06-13 20:49:15 +0000
+++ tests/vmtests/test_uefi_basic.py	2016-07-19 15:04:36 +0000
@@ -3,41 +3,15 @@
 from .releases import base_vm_classes as relbase
 
 import os
-import textwrap
-
-
-class TestBasicAbs(VMBaseClass):
+
+
+class TestUefiBasicAbs(VMBaseClass):
     interactive = False
-    arch_skip = ["s390x"]
+    arch_skip = ["s390x", "ppc64el"]
     conf_file = "examples/tests/uefi_basic.yaml"
     extra_disks = []
     uefi = True
     disk_to_check = [('main_disk', 1), ('main_disk', 2)]
-    collect_scripts = [textwrap.dedent("""
-        cd OUTPUT_COLLECT_D
-        blkid -o export /dev/vda > blkid_output_vda
-        blkid -o export /dev/vda1 > blkid_output_vda1
-        blkid -o export /dev/vda2 > blkid_output_vda2
-        cat /proc/partitions > proc_partitions
-        ls -al /dev/disk/by-uuid/ > ls_uuid
-        cat /etc/fstab > fstab
-        mkdir -p /dev/disk/by-dname
-        ls /dev/disk/by-dname/ > ls_dname
-        find /etc/network/interfaces.d > find_interfacesd
-        ls /sys/firmware/efi/ > ls_sys_firmware_efi
-        cat /sys/class/block/vda/queue/logical_block_size > vda_lbs
-        cat /sys/class/block/vda/queue/physical_block_size > vda_pbs
-        blockdev --getsz /dev/vda > vda_blockdev_getsz
-        blockdev --getss /dev/vda > vda_blockdev_getss
-        blockdev --getpbsz /dev/vda > vda_blockdev_getpbsz
-        blockdev --getbsz /dev/vda > vda_blockdev_getbsz
-        """)]
-
-    def test_output_files_exist(self):
-        self.output_files_exist(
-            ["blkid_output_vda", "blkid_output_vda1", "blkid_output_vda2",
-             "fstab", "ls_dname", "ls_uuid", "ls_sys_firmware_efi",
-             "proc_partitions"])
 
     def test_sys_firmware_efi(self):
         sys_efi_expected = [
@@ -61,11 +35,16 @@
         """ Test disk logical and physical block size are match
             the class block size.
         """
-        for bs in ['lbs', 'pbs']:
-            with open(os.path.join(self.td.collect,
-                      'vda_' + bs), 'r') as fp:
-                size = int(fp.read())
-                self.assertEqual(self.disk_block_size, size)
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'disk']:
+            vol_path = self.storage_volume_dict.get(id)
+            (vol_kname, ls_id) = self._resolve_volpath(vol_path,
+                                                       ls_input='ls_id')
+            for bs in ['lbs', 'pbs']:
+                with open(os.path.join(self.td.collect,
+                          bs + '_' + vol_kname), 'r') as fp:
+                    size = int(fp.read())
+                    self.assertEqual(self.disk_block_size, size)
 
     def test_disk_block_size_with_blockdev(self):
         """ validate maas setting
@@ -74,14 +53,19 @@
         --getpbsz                 get physical block (sector) size
         --getbsz                  get blocksize
         """
-        for syscall in ['getss', 'getpbsz']:
-            with open(os.path.join(self.td.collect,
-                      'vda_blockdev_' + syscall), 'r') as fp:
-                size = int(fp.read())
-                self.assertEqual(self.disk_block_size, size)
-
-
-class PreciseUefiTestBasic(relbase.precise, TestBasicAbs):
+        for id in [id for (id, item) in self.storage_config_dict.items()
+                   if item['type'] == 'disk']:
+            vol_path = self.storage_volume_dict.get(id)
+            (vol_kname, ls_id) = self._resolve_volpath(vol_path,
+                                                       ls_input='ls_id')
+            for syscall in ['getss', 'getpbsz']:
+                with open(os.path.join(self.td.collect,
+                          'blockdev_' + syscall + '_' + vol_kname), 'r') as fp:
+                    size = int(fp.read())
+                    self.assertEqual(self.disk_block_size, size)
+
+
+class PreciseUefiTestBasic(relbase.precise, TestUefiBasicAbs):
     __test__ = True
 
     def test_ptable(self):
@@ -91,7 +75,7 @@
         print("test_dname does not work for Precise")
 
 
-class TrustyUefiTestBasic(relbase.trusty, TestBasicAbs):
+class TrustyUefiTestBasic(relbase.trusty, TestUefiBasicAbs):
     __test__ = True
 
     # FIXME(LP: #1523037): dname does not work on trusty, so we cannot expect
@@ -104,15 +88,15 @@
         print("test_ptable does not work for Trusty")
 
 
-class WilyUefiTestBasic(relbase.wily, TestBasicAbs):
-    __test__ = True
-
-
-class XenialUefiTestBasic(relbase.xenial, TestBasicAbs):
-    __test__ = True
-
-
-class YakketyUefiTestBasic(relbase.yakkety, TestBasicAbs):
+class WilyUefiTestBasic(relbase.wily, TestUefiBasicAbs):
+    __test__ = True
+
+
+class XenialUefiTestBasic(relbase.xenial, TestUefiBasicAbs):
+    __test__ = True
+
+
+class YakketyUefiTestBasic(relbase.yakkety, TestUefiBasicAbs):
     __test__ = True
 
 

=== modified file 'tools/launch'
--- tools/launch	2016-06-27 14:01:39 +0000
+++ tools/launch	2016-07-19 15:04:36 +0000
@@ -8,7 +8,7 @@
 HTTP_PORT_MIN=${HTTP_PORT_MIN:-12000}
 HTTP_PORT_MAX=${HTTP_PORT_MAX:-65500}
 MY_D=$(dirname "$0")
-DEFAULT_ROOT_ARG="root=LABEL=cloudimg-rootfs"
+DEFAULT_ROOT_ARG="LABEL=cloudimg-rootfs"
 
 error() { echo "$@" 1>&2; }
 
@@ -578,13 +578,12 @@
         if [ -z "$root_arg" ]; then
             debug 1 "WARN: root_arg is empty with kernel."
         fi
-        append="${root_arg:+${root_arg} }ds=nocloud-net;seedfrom=$burl"
+        append="root=${root_arg:+${root_arg} }ds=nocloud-net;seedfrom=$burl"
 
-        local console_name=""
+        local console_name="ttyS0"
         case "${arch_hint}" in
-            s390x) console_name="";;
+            s390x) console_name="ttyS0";;
             ppc64*) console_name="hvc0";;
-            *) console_name="ttyS0";;
         esac
         if [ -n "$console_name" ]; then
             append="${append} console=${console_name}"
@@ -625,7 +624,7 @@
     cmd=(
         xkvm "${pt[@]}" "${netargs[@]}" --
         "${bios_opts[@]}"
-        -m ${mem} ${serial_args} ${video}
+        -smp 2 -m ${mem} ${serial_args} ${video}
         -drive "file=$bootimg,if=none,cache=unsafe,format=qcow2,id=boot,index=0"
         -device "virtio-blk,drive=boot"
         "${disk_args[@]}"