Merge lp:~smoser/curtin/merge-critical into lp:~curtin-dev/curtin/trunk

[Scott Moser]

this is a pull-ready merge of 3 branches.
hoping to test, see all magic pixies and commit.

[Ryan Harper (raharper)]

We're not installing filesystem deps in-target anymore (xfs_info not found):

Ideally the per-config dict of package mapping to config types is
what the deps/ stuff should be using; the fstype needs to check a second value
in the config to determine which filesystem was used.

Additionally, the current deps/__init__.py:install_deps doesn't expose target
which is available in the underlying util.install_packages, so in curthooks where
we really want to re-use deps.install_deps(), we want to pass it in target and
the custom config.

But the following allows test_mdadm_bcache.py to pass now.

% bzr diff curtin/commands/curthooks.py
=== modified file 'curtin/commands/curthooks.py'
--- curtin/commands/curthooks.py 2015-11-17 19:10:43 +0000
+++ curtin/commands/curthooks.py 2015-11-18 03:30:19 +0000
@@ -591,6 +591,12 @@
             'bridge-utils': ['bridge']},
     }

+ format_configs = {
+ 'xfsprogs': ['xfs'],
+ 'e2fsprogs': ['ext2', 'ext3', 'ext4'],
+ 'brtfs-tools': ['btrfs'],
+ }
+
     needed_packages = []
     installed_packages = get_installed_packages(target)
     for cust_cfg, pkg_reqs in custom_configs.items():
@@ -606,6 +612,15 @@
                pkg not in installed_packages:
                 needed_packages.append(pkg)

+ format_types = set(
+ [operation['fstype']
+ for operation in cfg[cust_cfg]['config']
+ if operation['type'] == 'format'])
+ for pkg, fstypes in format_configs.items():
+ if set(fstypes).intersection(format_types) and \
+ pkg not in installed_packages:
+ needed_packages.append(pkg)
+
     if needed_packages:
         util.install_packages(needed_packages, target=target)

review: Needs Fixing

[Ryan Harper (raharper)]

test_network.py:TrustyTestBasic fails due to missing python?-oauth package not installed.

Fixed with:

% bzr diff curtin/deps/__init__.py
=== modified file 'curtin/deps/__init__.py'
--- curtin/deps/__init__.py 2015-11-17 19:10:04 +0000
+++ curtin/deps/__init__.py 2015-11-18 06:09:41 +0000
@@ -23,6 +23,7 @@
 REQUIRED_IMPORTS = [
     # import string to execute, python2 package, python3 package
     ('import yaml', 'python-yaml', 'python3-yaml'),
+ ('import oauth', 'python-oauthlib', 'python3-oauthlib'),
 ]

 REQUIRED_EXECUTABLES = [

review: Needs Fixing

[Ryan Harper (raharper)]

% bzr diff curtin/deps/__init__.py
=== modified file 'curtin/deps/__init__.py'
--- curtin/deps/__init__.py 2015-11-17 19:10:04 +0000
+++ curtin/deps/__init__.py 2015-11-18 12:22:35 +0000
@@ -23,11 +23,13 @@
 REQUIRED_IMPORTS = [
     # import string to execute, python2 package, python3 package
     ('import yaml', 'python-yaml', 'python3-yaml'),
+ ('import oauth', 'python-oauthlib', 'python3-oauthlib'),
 ]

 REQUIRED_EXECUTABLES = [
     # executable in PATH, package
     ('lvcreate', 'lvm2'),
+ ('make-bcache', 'bcache-tools'),
     ('mdadm', 'mdadm'),
     ('mkfs.btrfs', 'btrfs-tools'),
     ('mkfs.ext4', 'e2fsprogs'),

And, I think this should have bcache-tools , not bcache-utils:

if lsb_release(field='codename') == "precise":
    REQUIRED_EXECUTABLES.append(('make-bcache', 'bcache-utils',))

[lp:~smoser/curtin/merge-critical]

302. By Scott Moser on 2015-11-18

mkfs.ext3 or mkfs.ext4 should be called with -F

There are times when mkfs.ext3 believes it knows better than you,
and will refuse to create a filesystem on a device.

I've not discovered the complete heuristics, but it seems
it might be related to the device name (vdb versus sdb), the contents
(filesystem or partition table or blank).

Since we are intending to mkfs, we should be passing -F, but
I am afraid that we're possibly masking some other issues here.
In that any device we're wiping we should be wiping any signatures
from the start and end of disk and partitions.
------------- This line and the following will be ignored --------------

modified:
  curtin/commands/block_meta.py
  curtin/commands/mkfs.py
pending merges:
  Scott Moser 2015-11-17 mkfs.ext3 or mkfs.ext4 should be called with -F

303. By Scott Moser on 2015-11-18

tests/vmtests/: do not remove tempdir on TempDir __del__

this was thwarting the attempts of CURTIN_VMTEST_KEEP_DATA
to keep data by deleting it anyway.

304. By Scott Moser on 2015-11-18

Allow re-use of bcache cache devices with separate backing devices

Decouple creation of bcache caches and backing devices by dropping
use of make-bcache -B <> -C <>, in favor of single use -B and -C
and combining backing devices with caches via sysfs attach attributes.
Update the mdadm/bcache vmtest to include a multi-bcache backing device
cached by single cachedevice test.

While validating results, fix lurking issue with mdadm.conf not getting
updated post raid configuration in target triggering LP: #964052.

305. By Scott Moser on 2015-11-18

Reduce chance udev race with mdadm opening /dev/mdX

In some cases mdadm races with udevd with the creation of /dev/mdX and
fails to open the newly created raid array. Reduce the window by stopping
the udev exec queue while creating the raid device and resuming it
afterwards. The attached testcase no longer reproduces the issue found
only on Trusty but may still lurk in Trusty mdadm (or older).

306. By Scott Moser on 2015-11-18

Improve dependency checking and installation

This makes several changes all toward the goal of fixing
curtin running on precise and better handing dependency installation.

 * change launcher to stop calling 'install-deps' itself, but rather just
   invoke the command it needs with the best python available
   the curtin main now handles installing its own dependencies if
   the user provides --install-deps.

 * pack_install now calls 'curtin pack-install --install-deps'

 * curtin/util and portions of curtin/commands/main.py are expected
   to be runnable with any python with only standard library modules.

Lastly, in order to verify all this is functional we add a Precise test.
That required some work to the test suite to handle powering the
system off because cloud-init in 12.04 does not have power_state module.

=== modified file 'bin/curtin'
--- bin/curtin	2015-03-09 16:35:00 +0000
+++ bin/curtin	2015-11-18 18:13:59 +0000
@@ -1,8 +1,7 @@
 #!/bin/sh
 PY3OR2_MAIN="curtin.commands.main"
 PY3OR2_MCHECK="curtin.deps.check"
-PY3OR2_MINSTALL="curtin.deps.install"
-PY3OR2_PYTHONS=${PY3OR2_PYTHONS:-"python3:python2:python"}
+PY3OR2_PYTHONS=${PY3OR2_PYTHONS:-"python3:python"}
 PYTHON=${PY3OR2_PYTHON}
 
 debug() {
@@ -27,25 +26,28 @@
 if [ ! -n "$PYTHON" ]; then
     first_exe=""
     oifs="$IFS"; IFS=":"
+    best=0
+    best_exe=""
     for p in $PY3OR2_PYTHONS; do
         command -v "$p" >/dev/null 2>&1 ||
             { debug "$p: not in path"; continue; }
+        [ -z "$PY3OR2_MCHECK" ] && PYTHON=$p && break
+        out=$($p -m "$PY3OR2_MCHECK" "$@" 2>&1) && PYTHON="$p" &&
+            { debug "$p passed check [$p -m $PY3OR2_MCHECK $*]"; break; }
+        ret=$?
+        debug "$p [$ret]: $out"
+        # exit code of 1 is unuseable
+        [ $ret -eq 1 ] && continue
         [ -n "$first_exe" ] || first_exe="$p"
-        [ -z "$PY3OR2_MCHECK" ] && PYTHON=$p && break
-        out=$($p -m "$PY3OR2_MCHECK" 2>&1) && PYTHON="$p" && break
-        debug "$p: $out"
+        # higher non-zero exit values indicate more plausible usability
+        [ $best -lt $ret ] && best_exe="$p" && best=$ret &&
+            debug "current best: $best_exe"
     done
     IFS="$oifs"
-    if [ -z "$PYTHON" ]; then
-        if [ -n "$first_exe" -a -n "$PY3OR2_MINSTALL" ]; then
-            debug "attempting deps install with $first_exe $PY3OR2_MINSTALL"
-            "$first_exe" -m "$PY3OR2_MINSTALL" ||
-                fail "failed to install deps!"
-            PYTHON="$first_exe"
-        fi
-    fi
+    [ -z "$best_exe" -a -n "$first_exe" ] && best_exe="$first_exe"
+    [ -n "$PYTHON" ] || PYTHON="$best_exe"
     [ -n "$PYTHON" ] ||
         fail "no availble python? [PY3OR2_DEBUG=1 for more info]"
 fi
-debug "executing: $PYTHON -m "$PY3OR2_MAIN" $*"
+debug "executing: $PYTHON -m \"$PY3OR2_MAIN\" $*"
 exec $PYTHON -m "$PY3OR2_MAIN" "$@"

=== modified file 'curtin/commands/block_meta.py'
--- curtin/commands/block_meta.py	2015-11-17 22:03:23 +0000
+++ curtin/commands/block_meta.py	2015-11-18 18:13:59 +0000
@@ -31,14 +31,12 @@
 import sys
 import tempfile
 import time
+import re
 
 SIMPLE = 'simple'
 SIMPLE_BOOT = 'simple-boot'
 CUSTOM = 'custom'
 
-CUSTOM_REQUIRED_PACKAGES = ['mdadm', 'lvm2', 'bcache-tools',
-                            'btrfs-tools', 'xfsprogs']
-
 CMD_ARGUMENTS = (
     ((('-D', '--devices'),
       {'help': 'which devices to operate on', 'action': 'append',
@@ -155,20 +153,43 @@
         util.subp(cmd, rcs=[0, 1, 2, 5], capture=True)
 
 
-def get_holders(devname):
+def block_find_sysfs_path(devname):
+    # Look up any block device holders.  Handle devices and partitions
+    # as devnames (vdb, md0, vdb7)
     if not devname:
         return []
 
-    devname_sysfs = \
-        '/sys/class/block/{}/holders'.format(os.path.basename(devname))
-    if not os.path.exists(devname_sysfs):
-        err = ('No sysfs path to device holders:'
+    sys_class_block = '/sys/class/block/'
+    basename = os.path.basename(devname)
+    # try without parent blockdevice, then prepend parent
+    paths = [
+        os.path.join(sys_class_block, basename),
+        os.path.join(sys_class_block,
+                     re.split('[\d+]', basename)[0], basename),
+    ]
+
+    # find path to devname directory in sysfs
+    devname_sysfs = None
+    for path in paths:
+        if os.path.exists(path):
+            devname_sysfs = path
+
+    if devname_sysfs is None:
+        err = ('No sysfs path to device:'
                ' {}'.format(devname_sysfs))
         LOG.error(err)
         raise ValueError(err)
 
-    LOG.debug('Getting blockdev holders: {}'.format(devname_sysfs))
-    return os.listdir(devname_sysfs)
+    return devname_sysfs
+
+
+def get_holders(devname):
+    devname_sysfs = block_find_sysfs_path(devname)
+    if devname_sysfs:
+        LOG.debug('Getting blockdev holders: {}'.format(devname_sysfs))
+        return os.listdir(os.path.join(devname_sysfs, 'holders'))
+
+    return []
 
 
 def clear_holders(sys_block_path):
@@ -892,7 +913,7 @@
     if mdname:
         mdnameparm = "--name=%s" % info.get('mdname')
 
-    cmd = ["yes", "|", "mdadm", "--create", "/dev/%s" % info.get('name'),
+    cmd = ["mdadm", "--create", "/dev/%s" % info.get('name'), "--run",
            "--level=%s" % raidlevel, "--raid-devices=%s" % len(device_paths),
            mdnameparm]
 
@@ -911,7 +932,11 @@
             cmd.append(device)
 
     # Create the raid device
+    util.subp(["udevadm", "settle"])
+    util.subp(["udevadm", "control", "--stop-exec-queue"])
     util.subp(" ".join(cmd), shell=True)
+    util.subp(["udevadm", "control", "--start-exec-queue"])
+    util.subp(["udevadm", "settle"])
 
     # Make dname rule for this dev
     make_dname(info.get('id'), storage_config)
@@ -945,17 +970,37 @@
     cache_mode = info.get('cache_mode', None)
 
     if not backing_device or not cache_device:
-        raise ValueError("backing device and cache device for bcache must be \
-                specified")
+        raise ValueError("backing device and cache device for bcache"
+                         " must be specified")
 
     # The bcache module is not loaded when bcache is installed by apt-get, so
     # we will load it now
     util.subp(["modprobe", "bcache"])
 
-    # If both the backing device and cache device are specified at the same
-    # time than it is not necessary to attach the cache device manually, as
-    # bcache will do this automatically.
-    util.subp(["make-bcache", "-B", backing_device, "-C", cache_device])
+    if cache_device:
+        # /sys/class/block/XXX/YYY/
+        cache_device_sysfs = block_find_sysfs_path(cache_device)
+
+        if os.path.exists(os.path.join(cache_device_sysfs, "bcache")):
+            # read in cset uuid from cache device
+            (out, err) = util.subp(["bcache-super-show", cache_device],
+                                   capture=True)
+            LOG.debug('out=[{}]'.format(out))
+            [cset_uuid] = [line.split()[-1] for line in out.split("\n")
+                           if line.startswith('cset.uuid')]
+
+        else:
+            # make the cache device, extracting cacheset uuid
+            (out, err) = util.subp(["make-bcache", "-C", cache_device],
+                                   capture=True)
+            LOG.debug('out=[{}]'.format(out))
+            [cset_uuid] = [line.split()[-1] for line in out.split("\n")
+                           if line.startswith('Set UUID:')]
+
+    if backing_device:
+        backing_device_sysfs = block_find_sysfs_path(backing_device)
+        if not os.path.exists(os.path.join(backing_device_sysfs, "bcache")):
+            util.subp(["make-bcache", "-B", backing_device])
 
     # Some versions of bcache-tools will register the bcache device as soon as
     # we run make-bcache using udev rules, so wait for udev to settle, then try
@@ -971,6 +1016,22 @@
             fp.write(path)
             fp.close()
 
+    # if we specify both then we need to attach backing to cache
+    if cache_device and backing_device:
+        if cset_uuid:
+            LOG.info("Attaching backing device to cacheset: "
+                     "{} -> {} cset.uuid: {}".format(backing_device,
+                                                     cache_device,
+                                                     cset_uuid))
+            attach = os.path.join(backing_device_sysfs,
+                                  "bcache",
+                                  "attach")
+            with open(attach, "w") as fp:
+                fp.write(cset_uuid)
+        else:
+            LOG.error("Invalid cset_uuid: {}".format(cset_uuid))
+            raise
+
     if cache_mode:
         # find the actual bcache device name via sysfs using the
         # backing device's holders directory.
@@ -999,19 +1060,6 @@
                          not supported")
 
 
-def install_missing_packages_for_meta_custom():
-    """Install all the missing package that `meta_custom` requires to
-    function properly."""
-    missing_packages = [
-        package
-        for package in CUSTOM_REQUIRED_PACKAGES
-        if not util.has_pkg_installed(package)
-    ]
-    if len(missing_packages) > 0:
-        util.apt_update()
-        util.install_packages(missing_packages)
-
-
 def meta_custom(args):
     """Does custom partitioning based on the layout provided in the config
     file. Section with the name storage contains information on which
@@ -1034,9 +1082,6 @@
     state = util.load_command_environment()
     cfg = config.load_command_config(args, state)
 
-    # make sure the required packages are installed
-    install_missing_packages_for_meta_custom()
-
     storage_config = cfg.get('storage', {})
     if not storage_config:
         raise Exception("storage configuration is required by mode '%s' "

=== modified file 'curtin/commands/curthooks.py'
--- curtin/commands/curthooks.py	2015-10-02 19:59:11 +0000
+++ curtin/commands/curthooks.py	2015-11-18 18:13:59 +0000
@@ -591,6 +591,12 @@
             'bridge-utils': ['bridge']},
     }
 
+    format_configs = {
+        'xfsprogs': ['xfs'],
+        'e2fsprogs': ['ext2', 'ext3', 'ext4'],
+        'brtfs-tools': ['btrfs'],
+    }
+
     needed_packages = []
     installed_packages = get_installed_packages(target)
     for cust_cfg, pkg_reqs in custom_configs.items():
@@ -606,6 +612,15 @@
                pkg not in installed_packages:
                 needed_packages.append(pkg)
 
+        format_types = set(
+            [operation['fstype']
+             for operation in cfg[cust_cfg]['config']
+             if operation['type'] == 'format'])
+        for pkg, fstypes in format_configs.items():
+            if set(fstypes).intersection(format_types) and \
+               pkg not in installed_packages:
+                needed_packages.append(pkg)
+
     if needed_packages:
         util.install_packages(needed_packages, target=target)
 
@@ -687,6 +702,10 @@
                                   "mdadm.conf")
     if os.path.exists(mdadm_location):
         copy_mdadm_conf(mdadm_location, target)
+        # as per https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/964052
+        # reconfigure mdadm
+        util.subp(['chroot', target, 'dpkg-reconfigure',
+                   '--frontend=noninteractive', 'mdadm'], data=None)
 
     # If udev dname rules were created, copy them to target
     udev_rules_d = os.path.join(state['scratch'], "rules.d")

=== modified file 'curtin/commands/main.py'
--- curtin/commands/main.py	2015-10-20 16:56:30 +0000
+++ curtin/commands/main.py	2015-11-18 18:13:59 +0000
@@ -23,8 +23,7 @@
 
 from .. import log
 from .. import util
-from .. import config
-from ..reporter import (events, update_configuration)
+from ..deps import install_deps
 
 SUB_COMMAND_MODULES = ['apply_net', 'block-meta', 'curthooks', 'extract',
                        'hook', 'in-target', 'install', 'mkfs', 'net-meta',
@@ -43,20 +42,20 @@
     popfunc(subparser.add_parser(subcmd))
 
 
-def main(args=None):
-    if args is None:
-        args = sys.argv
-
-    parser = argparse.ArgumentParser()
-
-    stacktrace = (os.environ.get('CURTIN_STACKTRACE', "0").lower()
-                  not in ("0", "false", ""))
-
-    try:
-        verbosity = int(os.environ.get('CURTIN_VERBOSITY', "0"))
-    except ValueError:
-        verbosity = 1
-
+class NoHelpParser(argparse.ArgumentParser):
+    # ArgumentParser with forced 'add_help=False'
+    def __init__(self, *args, **kwargs):
+        kwargs.update({'add_help': False})
+        super(NoHelpParser, self).__init__(*args, **kwargs)
+
+    def error(self, message):
+        # without overriding this, argparse exits with bad usage
+        raise ValueError("failed parsing arguments: %s" % message)
+
+
+def get_main_parser(stacktrace=False, verbosity=0,
+                    parser_class=argparse.ArgumentParser):
+    parser = parser_class(prog='curtin')
     parser.add_argument('--showtrace', action='store_true', default=stacktrace)
     parser.add_argument('-v', '--verbose', action='count', default=verbosity,
                         dest='verbosity')
@@ -66,6 +65,9 @@
                         help='read configuration from cfg',
                         metavar='FILE', type=argparse.FileType("rb"),
                         dest='main_cfgopts', default=[])
+    parser.add_argument('--install-deps', action='store_true',
+                        help='install dependencies as necessary',
+                        default=False)
     parser.add_argument('--set', action=util.MergedCmdAppend,
                         help=('define a config variable. key can be a "/" '
                               'delimited path ("early_commands/cmd1=a"). if '
@@ -75,6 +77,66 @@
     parser.set_defaults(config={})
     parser.set_defaults(reportstack=None)
 
+    return parser
+
+
+def maybe_install_deps(args, stacktrace=True, verbosity=0):
+    parser = get_main_parser(stacktrace=stacktrace, verbosity=verbosity,
+                             parser_class=NoHelpParser)
+    subps = parser.add_subparsers(dest="subcmd", parser_class=NoHelpParser)
+    for subcmd in SUB_COMMAND_MODULES:
+        subps.add_parser(subcmd)
+
+    install_only_args = [
+       ['-v', '--install-deps'],
+       ['-vv', '--install-deps'],
+       ['--install-deps', '-v'],
+       ['--install-deps', '-vv'],
+       ['--install-deps'],
+    ]
+
+    install_only = args in install_only_args
+     
+    if install_only:
+        verbosity = 1
+    else:
+        try:
+            ns, unknown = parser.parse_known_args(args)
+            verbosity = ns.verbosity
+            if not ns.install_deps:
+                return
+        except ValueError as e:
+            # bad usage will be reported by the real reporter
+            return
+
+    ret = install_deps(verbosity=verbosity)
+
+    if ret != 0 or install_only:
+        sys.exit(ret)
+
+    return
+
+
+def main(args=None):
+    if args is None:
+        args = sys.argv
+
+    stacktrace = (os.environ.get('CURTIN_STACKTRACE', "0").lower()
+                  not in ("0", "false", ""))
+
+    try:
+        verbosity = int(os.environ.get('CURTIN_VERBOSITY', "0"))
+    except ValueError:
+        verbosity = 1
+
+    maybe_install_deps(sys.argv[1:], stacktrace=stacktrace,
+                       verbosity=verbosity)
+
+    # Above here, only standard library modules can be assumed.
+    from .. import config
+    from ..reporter import (events, update_configuration)
+
+    parser = get_main_parser(stacktrace=stacktrace, verbosity=verbosity)
     subps = parser.add_subparsers(dest="subcmd")
     for subcmd in SUB_COMMAND_MODULES:
         add_subcmd(subps, subcmd)

=== modified file 'curtin/deps/__init__.py'
--- curtin/deps/__init__.py	2015-03-06 22:06:40 +0000
+++ curtin/deps/__init__.py	2015-11-18 18:13:59 +0000
@@ -1,1 +1,163 @@
+#   Copyright (C) 2015 Canonical Ltd.
+#
+#   Author: Scott Moser <[e-mail]>
+#
+#   Curtin is free software: you can redistribute it and/or modify it under
+#   the terms of the GNU Affero General Public License as published by the
+#   Free Software Foundation, either version 3 of the License, or (at your
+#   option) any later version.
+#
+#   Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
+#   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+#   FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for
+#   more details.
+#
+#   You should have received a copy of the GNU Affero General Public License
+#   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
+import os
+import sys
+
+from curtin.util import (which, install_packages, lsb_release,
+                         ProcessExecutionError)
+
+REQUIRED_IMPORTS = [
+    # import string to execute, python2 package, python3 package
+    ('import yaml', 'python-yaml', 'python3-yaml'),
+]
+
+REQUIRED_EXECUTABLES = [
+    # executable in PATH, package
+    ('lvcreate', 'lvm2'),
+    ('mdadm', 'mdadm'),
+    ('mkfs.btrfs', 'btrfs-tools'),
+    ('mkfs.ext4', 'e2fsprogs'),
+    ('mkfs.xfs', 'xfsprogs'),
+    ('partprobe', 'parted'),
+    ('sgdisk', 'gdisk'),
+    ('udevadm', 'udev'),
+]
+
+if lsb_release()['codename'] == "precise":
+    REQUIRED_IMPORTS.append(
+        ('import oauth.oauth', 'python-oauth', None),)
+else:
+    REQUIRED_EXECUTABLES.append(('make-bcache', 'bcache-tools',))
+    REQUIRED_IMPORTS.append(
+        ('import oauthlib.oauth1', 'python-oauthlib', 'python3-oauthlib'),)
+
+
+class MissingDeps(Exception):
+    def __init__(self, message, deps):
+        self.message = message
+        if isinstance(deps, str) or deps is None:
+            deps = [deps]
+        self.deps = [d for d in deps if d is not None]
+        self.fatal = None in deps
+
+    def __str__(self):
+        if self.fatal:
+            if not len(self.deps):
+                return self.message + " Unresolvable."
+            return (self.message +
+                    " Unresolvable.  Partially resolvable with packages: %s" %
+                    ' '.join(self.deps))
+        else:
+            return self.message + " Install packages: %s" % ' '.join(self.deps)
+
+
+def check_import(imports, py2pkgs, py3pkgs, message=None):
+    import_group = imports
+    if isinstance(import_group, str):
+        import_group = [import_group]
+
+    for istr in import_group:
+        try:
+            exec(istr)
+            return
+        except ImportError:
+            pass
+
+    if not message:
+        if isinstance(imports, str):
+            message = "Failed '%s'." % imports
+        else:
+            message = "Unable to do any of %s." % import_group
+
+    if sys.version_info[0] == 2:
+        pkgs = py2pkgs
+    else:
+        pkgs = py3pkgs
+
+    raise MissingDeps(message, pkgs)
+
+
+def check_executable(cmdname, pkg):
+    if not which(cmdname):
+        raise MissingDeps("Missing program '%s'." % cmdname, pkg)
+
+
+def check_executables(executables=None):
+    if executables is None:
+        executables = REQUIRED_EXECUTABLES
+    mdeps = []
+    for exe, pkg in executables:
+        try:
+            check_executable(exe, pkg)
+        except MissingDeps as e:
+            mdeps.append(e)
+    return mdeps
+
+
+def check_imports(imports=None):
+    if imports is None:
+        imports = REQUIRED_IMPORTS
+
+    mdeps = []
+    for import_str, py2pkg, py3pkg in imports:
+        try:
+            check_import(import_str, py2pkg, py3pkg)
+        except MissingDeps as e:
+            mdeps.append(e)
+    return mdeps
+
+
+def find_missing_deps():
+    return check_executables() + check_imports()
+
+
+def install_deps(verbosity=False, dry_run=False, allow_daemons=True):
+    errors = find_missing_deps()
+    if len(errors) == 0:
+        if verbosity:
+            sys.stderr.write("No missing dependencies\n")
+        return 0
+
+    missing_pkgs = []
+    for e in errors:
+        missing_pkgs += e.deps
+
+    deps_string = ' '.join(sorted(missing_pkgs))
+
+    if dry_run:
+        sys.stderr.write("Missing dependencies: %s\n" % deps_string)
+        return 0
+
+    if os.geteuid() != 0:
+        sys.stderr.write("Missing dependencies: %s\n" % deps_string)
+        sys.stderr.write("Package installation is not possible as non-root.\n")
+        return 2
+
+    if verbosity:
+        sys.stderr.write("Installing %s\n" % deps_string)
+
+    ret = 0
+    try:
+        install_packages(missing_pkgs, allow_daemons=allow_daemons)
+    except ProcessExecutionError as e:
+        sys.stderr.write("%s\n" % e)
+        ret = e.exit_code
+
+    return ret
+
+
 # vi: ts=4 expandtab syntax=python

=== modified file 'curtin/deps/check.py'
--- curtin/deps/check.py	2015-08-10 14:33:03 +0000
+++ curtin/deps/check.py	2015-11-18 18:13:59 +0000
@@ -20,29 +20,57 @@
 and exit success or fail, indicating that deps should be there.
   python -m curtin.deps.check [-v]
 """
-_imports = (
-    "from ..commands import main",
-    "import yaml",
-)
-
-
-def _check_imports(imports=_imports):
-    errors = []
-    for istr in _imports:
-        try:
-            exec(istr)
-        except ImportError as e:
-            errors.append("failed '%s': %s" % (istr, e))
-
-    return errors
+import argparse
+import sys
+
+from . import find_missing_deps
+
+
+def debug(level, msg_level, msg):
+    if level >= msg_level:
+        if msg[-1] != "\n":
+            msg += "\n"
+        sys.stderr.write(msg)
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog='curtin-check-deps',
+        description='check dependencies for curtin.')
+    parser.add_argument('-v', '--verbose', action='count', default=0,
+                        dest='verbosity')
+    args, extra = parser.parse_known_args(sys.argv[1:])
+
+    errors = find_missing_deps()
+
+    if len(errors) == 0:
+        # exit 0 means all dependencies are available.
+        debug(args.verbosity, 1, "No missing dependencies")
+        sys.exit(0)
+
+    missing_pkgs = []
+    fatal = []
+    for e in errors:
+        if e.fatal:
+            fatal.append(e)
+        debug(args.verbosity, 2, str(e))
+        missing_pkgs += e.deps
+
+    if len(fatal):
+        for e in fatal:
+            debug(args.verbosity, 1, str(e))
+        sys.exit(1)
+
+    debug(args.verbosity, 1,
+          "Fix with:\n  apt-get -qy install %s\n" %
+          ' '.join(sorted(missing_pkgs)))
+
+    # we exit higher with less deps needed.
+    # exiting 99 means just 1 dep needed.
+    sys.exit(100-len(missing_pkgs))
+
 
 if __name__ == '__main__':
-    import sys
-    verbose = False
-    if len(sys.argv) > 1 and sys.argv[1] in ("-v", "--verbose"):
-        verbose = True
-    errors = _check_imports()
-    if verbose:
-        for emsg in errors:
-            sys.stderr.write("%s\n" % emsg)
-    sys.exit(len(errors))
+    main()
+
+# vi: ts=4 expandtab syntax=python

=== modified file 'curtin/deps/install.py'
--- curtin/deps/install.py	2015-08-10 14:33:03 +0000
+++ curtin/deps/install.py	2015-11-18 18:13:59 +0000
@@ -20,33 +20,29 @@
   python -m curtin.deps.install [-v]
 """
 
-import subprocess
+import argparse
 import sys
-import time
-
-
-def runcmd(cmd, retries=[]):
-    for wait in retries:
-        try:
-            subprocess.check_call(cmd)
-            return 0
-        except subprocess.CalledProcessError as e:
-            sys.stderr.write("%s failed. sleeping %s\n" % (cmd, wait))
-            time.sleep(wait)
-    try:
-        subprocess.check_call(cmd)
-    except subprocess.CalledProcessError as e:
-        return e.returncode
+
+from . import install_deps
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog='curtin-install-deps',
+        description='install dependencies for curtin.')
+    parser.add_argument('-v', '--verbose', action='count', default=0,
+                        dest='verbosity')
+    parser.add_argument('--dry-run', action='store_true', default=False)
+    parser.add_argument('--no-allow-daemons', action='store_false',
+                        default=True)
+    args = parser.parse_args(sys.argv[1:])
+
+    ret = install_deps(verbose=args.verbosity, dry_run=args.dry_run,
+                       allow_daemons=True)
+    sys.exit(ret)
+
 
 if __name__ == '__main__':
-    if sys.version_info[0] == 2:
-        pkgs = ['python-yaml']
-    else:
-        pkgs = ['python3-yaml']
-
-    ret = runcmd(['apt-get', '--quiet', 'update'], retries=[2, 4])
-    if ret != 0:
-        sys.exit(ret)
-
-    ret = runcmd(['apt-get', 'install', '--quiet', '--assume-yes'] + pkgs)
-    sys.exit(ret)
+    main()
+
+# vi: ts=4 expandtab syntax=python

=== modified file 'curtin/pack.py'
--- curtin/pack.py	2015-03-10 17:17:55 +0000
+++ curtin/pack.py	2015-11-18 18:13:59 +0000
@@ -26,7 +26,6 @@
 #!/bin/sh
 PY3OR2_MAIN="%(ep_main)s"
 PY3OR2_MCHECK="%(ep_mcheck)s"
-PY3OR2_MINSTALL="%(ep_minstall)s"
 PY3OR2_PYTHONS=${PY3OR2_PYTHONS:-"%(python_exe_list)s"}
 PYTHON=${PY3OR2_PYTHON}
 """.strip()
@@ -54,41 +53,42 @@
 if [ ! -n "$PYTHON" ]; then
     first_exe=""
     oifs="$IFS"; IFS=":"
+    best=0
+    best_exe=""
     for p in $PY3OR2_PYTHONS; do
         command -v "$p" >/dev/null 2>&1 ||
             { debug "$p: not in path"; continue; }
+        [ -z "$PY3OR2_MCHECK" ] && PYTHON=$p && break
+        out=$($p -m "$PY3OR2_MCHECK" "$@" 2>&1) && PYTHON="$p" &&
+            { debug "$p passed check [$p -m $PY3OR2_MCHECK $*]"; break; }
+        ret=$?
+        debug "$p [$ret]: $out"
+        # exit code of 1 is unuseable
+        [ $ret -eq 1 ] && continue
         [ -n "$first_exe" ] || first_exe="$p"
-        [ -z "$PY3OR2_MCHECK" ] && PYTHON=$p && break
-        out=$($p -m "$PY3OR2_MCHECK" 2>&1) && PYTHON="$p" && break
-        debug "$p: $out"
+        # higher non-zero exit values indicate more plausible usability
+        [ $best -lt $ret ] && best_exe="$p" && best=$ret &&
+            debug "current best: $best_exe"
     done
     IFS="$oifs"
-    if [ -z "$PYTHON" ]; then
-        if [ -n "$first_exe" -a -n "$PY3OR2_MINSTALL" ]; then
-            debug "attempting deps install with $first_exe $PY3OR2_MINSTALL"
-            "$first_exe" -m "$PY3OR2_MINSTALL" ||
-                fail "failed to install deps!"
-            PYTHON="$first_exe"
-        fi
-    fi
+    [ -z "$best_exe" -a -n "$first_exe" ] && best_exe="$first_exe"
+    [ -n "$PYTHON" ] || PYTHON="$best_exe"
     [ -n "$PYTHON" ] ||
         fail "no availble python? [PY3OR2_DEBUG=1 for more info]"
 fi
-debug "executing: $PYTHON -m \"$PY3OR2_MAIN\" $*"
+debug "executing: $PYTHON -m \\"$PY3OR2_MAIN\\" $*"
 exec $PYTHON -m "$PY3OR2_MAIN" "$@"
 """
 
 
 def write_exe_wrapper(entrypoint, path=None, interpreter=None,
-                      deps_check_entry=None, deps_install_entry=None,
-                      mode=0o755):
+                      deps_check_entry=None, mode=0o755):
     if not interpreter:
-        interpreter = "python3:python2:python"
+        interpreter = "python3:python"
 
     subs = {
         'ep_main': entrypoint,
         'ep_mcheck': deps_check_entry if deps_check_entry else "",
-        'ep_minstall': deps_install_entry if deps_install_entry else "",
         'python_exe_list': interpreter,
     }
 
@@ -140,9 +140,7 @@
                         ignore=not_dot_py)
         write_exe_wrapper(entrypoint='curtin.commands.main',
                           path=os.path.join(bindir, 'curtin'),
-                          deps_check_entry="curtin.deps.check",
-                          deps_install_entry="curtin.deps.install"
-                          )
+                          deps_check_entry="curtin.deps.check")
 
         for archpath, filepath in copy_files:
             target = os.path.abspath(os.path.join(exdir, archpath))
@@ -208,7 +206,8 @@
 
 
 def pack_install(fdout=None, configs=None, paths=None,
-                 add_files=None, copy_files=None, args=None):
+                 add_files=None, copy_files=None, args=None,
+                 install_deps=True):
 
     if configs is None:
         configs = []
@@ -219,7 +218,12 @@
     if args is None:
         args = []
 
-    command = ["curtin", "install"]
+    if install_deps:
+        dep_flags = ["--install-deps"]
+    else:
+        dep_flags = []
+
+    command = ["curtin", dep_flags, "install"]
 
     my_files = []
     for n, config in enumerate(configs):

=== modified file 'curtin/util.py'
--- curtin/util.py	2015-10-26 18:03:20 +0000
+++ curtin/util.py	2015-11-18 18:13:59 +0000
@@ -30,6 +30,8 @@
 _INSTALLED_HELPERS_PATH = '/usr/lib/curtin/helpers'
 _INSTALLED_MAIN = '/usr/bin/curtin'
 
+_LSB_RELEASE = {}
+
 
 def _subp(args, data=None, rcs=None, env=None, capture=False, shell=False,
           logstring=False):
@@ -691,6 +693,31 @@
     return (isinstance(exc, IOError) and exc.errno == errno.ENOENT)
 
 
+def lsb_release():
+    fmap = {'Codename': 'codename', 'Description': 'description',
+            'Distributor ID': 'id', 'Release': 'release'}
+    global _LSB_RELEASE
+    if not _LSB_RELEASE:
+        data = {}
+        try:
+            out, err = subp(['lsb_release', '--all'], capture=True)
+            for line in out.splitlines():
+                fname, tok, val = line.partition(":")
+                if fname in fmap:
+                    data[fmap[fname]] = val.strip()
+            missing = [k for k in fmap.values() if k not in data]
+            if len(missing):
+                LOG.warn("Missing fields in lsb_release --all output: %s",
+                         ','.join(missing))
+
+        except ProcessExecutionError as e:
+            LOG.warn("Unable to get lsb_release --all: %s", e)
+            data = {v: "UNAVAILABLE" for v in fmap.values()}
+
+        _LSB_RELEASE.update(data)
+    return _LSB_RELEASE
+
+
 class MergedCmdAppend(argparse.Action):
     """This appends to a list in order of appearence both the option string
        and the value"""

=== modified file 'debian/control'
--- debian/control	2015-10-07 13:44:24 +0000
+++ debian/control	2015-11-18 18:13:59 +0000
@@ -25,7 +25,15 @@
 Package: curtin
 Architecture: all
 Priority: extra
-Depends: python3-curtin (= ${binary:Version}),
+Depends: btrfs-tools,
+         e2fsprogs,
+         gdisk,
+         lvm2,
+         mdadm,
+         parted,
+         python3-curtin (= ${binary:Version}),
+         udev,
+         xfsprogs,
          ${misc:Depends}
 Description: Library and tools for the curtin installer
  This package provides the curtin installer.

=== modified file 'examples/tests/mdadm_bcache.yaml'
--- examples/tests/mdadm_bcache.yaml	2015-10-29 19:13:25 +0000
+++ examples/tests/mdadm_bcache.yaml	2015-11-18 18:13:59 +0000
@@ -32,6 +32,10 @@
        type: partition
        size: 1GB
        device: sda
+     - id: sda6
+       type: partition
+       size: 1GB
+       device: sda
      - id: mddevice
        name: md0
        type: raid
@@ -47,6 +51,12 @@
        backing_device: mddevice
        cache_device: sda5
        cache_mode: writeback
+     - id: bcache1
+       type: bcache
+       name: cached_array_2
+       backing_device: sda6
+       cache_device: sda5
+       cache_mode: writeback
      - id: sda1_root
        type: format
        fstype: ext4
@@ -55,6 +65,10 @@
        type: format
        fstype: ext4
        volume: bcache0
+     - id: bcache_storage
+       type: format
+       fstype: ext4
+       volume: bcache1
      - id: sda1_mount
        type: mount
        path: /
@@ -63,3 +77,7 @@
        type: mount
        path: /media/data
        device: raid_storage
+     - id: bcache1_mount
+       type: mount
+       path: /media/bcache1
+       device: bcache_storage

=== added file 'examples/tests/raid5bcache.yaml'
--- examples/tests/raid5bcache.yaml	1970-01-01 00:00:00 +0000
+++ examples/tests/raid5bcache.yaml	2015-11-18 18:13:59 +0000
@@ -0,0 +1,97 @@
+storage:
+  config:
+  - grub_device: true
+    id: sda
+    model: QEMU HARDDISK
+    name: sda
+    ptable: msdos
+    path: /dev/vdb
+    type: disk
+    wipe: superblock
+  - id: sdb
+    model: QEMU HARDDISK
+    name: sdb
+    path: /dev/vdc
+    type: disk
+    wipe: superblock
+  - id: sdc
+    model: QEMU HARDDISK
+    name: sdc
+    path: /dev/vdd
+    type: disk
+    wipe: superblock
+  - id: sdd
+    model: QEMU HARDDISK
+    name: sdd
+    path: /dev/vde
+    type: disk
+    wipe: superblock
+  - id: sde
+    model: QEMU HARDDISK
+    name: sde
+    path: /dev/vdf
+    type: disk
+    wipe: superblock
+  - devices:
+    - sdc
+    - sdd
+    - sde
+    id: md0
+    name: md0
+    raidlevel: 5
+    spare_devices: []
+    type: raid
+  - device: sda
+    id: sda-part1
+    name: sda-part1
+    number: 1
+    offset: 2097152B
+    size: 1000001536B
+    type: partition
+    uuid: 3a38820c-d675-4069-b060-509a3d9d13cc
+    wipe: superblock
+  - device: sda
+    id: sda-part2
+    name: sda-part2
+    number: 2
+    size: 7586787328B
+    type: partition
+    uuid: 17747faa-4b9e-4411-97e5-12fd3d199fb8
+    wipe: superblock
+  - backing_device: sda-part2
+    cache_device: sdb
+    cache_mode: writeback
+    id: bcache0
+    name: bcache0
+    type: bcache
+  - fstype: ext4
+    id: sda-part1_format
+    label: ''
+    type: format
+    uuid: 71b1ef6f-5cab-4a77-b4c8-5a209ec11d7c
+    volume: sda-part1
+  - fstype: ext4
+    id: md0_format
+    label: ''
+    type: format
+    uuid: b031f0a0-adb3-43be-bb43-ce0fc8a224a4
+    volume: md0
+  - fstype: ext4
+    id: bcache0_format
+    label: ''
+    type: format
+    uuid: ce45bbaf-5a44-4487-b89e-035c2dd40657
+    volume: bcache0
+  - device: bcache0_format
+    id: bcache0_mount
+    path: /
+    type: mount
+  - device: sda-part1_format
+    id: sda-part1_mount
+    path: /boot
+    type: mount
+  - device: md0_format
+    id: md0_mount
+    path: /srv/data
+    type: mount
+  version: 1

=== modified file 'tests/unittests/test_util.py'
--- tests/unittests/test_util.py	2014-09-18 01:05:55 +0000
+++ tests/unittests/test_util.py	2015-11-18 18:13:59 +0000
@@ -1,4 +1,5 @@
 from unittest import TestCase
+import mock
 import os
 import shutil
 import tempfile
@@ -99,6 +100,45 @@
         self.assertEqual(found, "/usr/bin2/fuzz")
 
 
+class TestLsbRelease(TestCase):
+    def setUp(self):
+        self._reset_cache()
+
+    def _reset_cache(self):
+        keys = [k for k in util._LSB_RELEASE.keys()]
+        for d in keys:
+            del util._LSB_RELEASE[d]
+
+    @mock.patch("curtin.util.subp")
+    def test_lsb_release_functional(self, mock_subp):
+        output = '\n'.join([
+            "Distributor ID: Ubuntu",
+            "Description:    Ubuntu 14.04.2 LTS",
+            "Release:    14.04",
+            "Codename:   trusty",
+        ])
+        rdata = {'id': 'Ubuntu', 'description': 'Ubuntu 14.04.2 LTS',
+                 'codename': 'trusty', 'release': '14.04'}
+
+        def fake_subp(cmd, capture=False):
+            return output, 'No LSB modules are available.'
+
+        mock_subp.side_effect = fake_subp
+        found = util.lsb_release()
+        mock_subp.assert_called_with(['lsb_release', '--all'], capture=True)
+        self.assertEqual(found, rdata)
+
+    @mock.patch("curtin.util.subp")
+    def test_lsb_release_unavailable(self, mock_subp):
+        def doraise(*args, **kwargs):
+            raise util.ProcessExecutionError("foo")
+        mock_subp.side_effect = doraise
+
+        expected = {k: "UNAVAILABLE" for k in
+                    ('id', 'description', 'codename', 'release')}
+        self.assertEqual(util.lsb_release(), expected)
+
+
 class TestSubp(TestCase):
 
     def test_subp_handles_utf8(self):

=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py	2015-11-18 16:24:35 +0000
+++ tests/vmtests/__init__.py	2015-11-18 18:13:59 +0000
@@ -483,7 +483,15 @@
     collect_post = textwrap.dedent(
         'tar -C "%s" -cf "%s" .' % (output_dir, output_device))
 
-    scripts = [collect_prep] + collect_scripts + [collect_post]
+    # failsafe poweroff runs on precise only, where power_state does
+    # not exist.
+    precise_poweroff = textwrap.dedent("""#!/bin/sh
+        [ "$(lsb_release -sc)" = "precise" ] || exit 0;
+        shutdown -P now "Shutting down on precise"
+        """)
+
+    scripts = ([collect_prep] + collect_scripts + [collect_post] +
+               [precise_poweroff])
 
     for part in scripts:
         if not part.startswith("#!"):

=== modified file 'tests/vmtests/test_basic.py'
--- tests/vmtests/test_basic.py	2015-10-27 11:35:10 +0000
+++ tests/vmtests/test_basic.py	2015-11-18 18:13:59 +0000
@@ -23,7 +23,10 @@
         cat /etc/fstab > fstab
         ls /dev/disk/by-dname/ > ls_dname
 
-        apt-config dump Acquire::HTTP::Proxy > apt-proxy
+        v=""
+        out=$(apt-config shell v Acquire::HTTP::Proxy)
+        eval "$out"
+        echo "$v" > apt-proxy
         """)]
 
     def test_output_files_exist(self):
@@ -67,7 +70,7 @@
     def test_proxy_set(self):
         expected = get_apt_proxy()
         with open(os.path.join(self.td.mnt, "apt-proxy")) as fp:
-            apt_proxy_found = fp.read()
+            apt_proxy_found = fp.read().rstrip()
         if expected:
             # the proxy should have gotten set through
             self.assertIn(expected, apt_proxy_found)
@@ -88,3 +91,16 @@
     repo = "maas-daily"
     release = "vivid"
     arch = "amd64"
+
+
+class PreciseTestBasic(TestBasicAbs, TestCase):
+    __test__ = True
+    repo = "maas-daily"
+    release = "precise"
+    arch = "amd64"
+
+    def test_ptable(self):
+        print("test_ptable does not work for Precise")
+
+    def test_dname(self):
+        print("test_dname does not work for Precise")

=== modified file 'tests/vmtests/test_mdadm_bcache.py'
--- tests/vmtests/test_mdadm_bcache.py	2015-10-30 13:38:36 +0000
+++ tests/vmtests/test_mdadm_bcache.py	2015-11-18 18:13:59 +0000
@@ -44,31 +44,60 @@
                      'main_disk': 5,
                      'main_disk': 6,
                      'md0': 0,
-                     'cached_array': 0}
+                     'cached_array': 0,
+                     'cached_array_2': 0}
 
     collect_scripts = TestMdadmAbs.collect_scripts + [textwrap.dedent("""
         cd OUTPUT_COLLECT_D
         bcache-super-show /dev/vda6 > bcache_super_vda6
+        bcache-super-show /dev/vda7 > bcache_super_vda7
+        bcache-super-show /dev/md0 > bcache_super_md0
         ls /sys/fs/bcache > bcache_ls
         cat /sys/block/bcache0/bcache/cache_mode > bcache_cache_mode
+        cat /proc/mounts > proc_mounts
         """)]
     fstab_expected = {
-        '/dev/bcache0': '/media/data'
+        '/dev/bcache0': '/media/data',
+        '/dev/bcache1': '/media/bcache1',
     }
 
     def test_bcache_output_files_exist(self):
-        self.output_files_exist(["bcache_super_vda6", "bcache_ls",
+        self.output_files_exist(["bcache_super_vda6",
+                                 "bcache_super_vda7",
+                                 "bcache_super_md0",
+                                 "bcache_ls",
                                  "bcache_cache_mode"])
 
     def test_bcache_status(self):
+        bcache_supers = [
+            "bcache_super_vda6",
+            "bcache_super_vda7",
+            "bcache_super_md0",
+        ]
         bcache_cset_uuid = None
-        with open(os.path.join(self.td.mnt, "bcache_super_vda6"), "r") as fp:
-            for line in fp.read().splitlines():
-                if line != "" and line.split()[0] == "cset.uuid":
-                    bcache_cset_uuid = line.split()[-1].rstrip()
-        self.assertIsNotNone(bcache_cset_uuid)
-        with open(os.path.join(self.td.mnt, "bcache_ls"), "r") as fp:
-            self.assertTrue(bcache_cset_uuid in fp.read().splitlines())
+        found = {}
+        for bcache_super in bcache_supers:
+            with open(os.path.join(self.td.mnt, bcache_super), "r") as fp:
+                for line in fp.read().splitlines():
+                    if line != "" and line.split()[0] == "cset.uuid":
+                        bcache_cset_uuid = line.split()[-1].rstrip()
+                        if bcache_cset_uuid in found:
+                            found[bcache_cset_uuid].append(bcache_super)
+                        else:
+                            found[bcache_cset_uuid] = [bcache_super]
+            self.assertIsNotNone(bcache_cset_uuid)
+            with open(os.path.join(self.td.mnt, "bcache_ls"), "r") as fp:
+                self.assertTrue(bcache_cset_uuid in fp.read().splitlines())
+
+        # one cset.uuid for all devices
+        self.assertEqual(len(found), 1)
+
+        # three devices with same cset.uuid
+        self.assertEqual(len(found[bcache_cset_uuid]), 3)
+
+        # check the cset.uuid in the dict
+        self.assertEqual(list(found.keys()).pop(),
+                         bcache_cset_uuid)
 
     def test_bcache_cachemode(self):
         self.check_file_regex("bcache_cache_mode", r"\[writeback\]")

=== added file 'tests/vmtests/test_raid5_bcache.py'
--- tests/vmtests/test_raid5_bcache.py	1970-01-01 00:00:00 +0000
+++ tests/vmtests/test_raid5_bcache.py	2015-11-18 18:13:59 +0000
@@ -0,0 +1,85 @@
+from . import VMBaseClass
+from unittest import TestCase
+
+import textwrap
+import os
+
+
+class TestMdadmAbs(VMBaseClass, TestCase):
+    __test__ = False
+    repo = "maas-daily"
+    arch = "amd64"
+    install_timeout = 600
+    boot_timeout = 100
+    interactive = False
+    extra_disks = ['10G', '10G', '10G', '10G']
+    active_mdadm = "1"
+    collect_scripts = [textwrap.dedent("""
+        cd OUTPUT_COLLECT_D
+        cat /etc/fstab > fstab
+        mdadm --detail --scan > mdadm_status
+        mdadm --detail --scan | grep -c ubuntu > mdadm_active1
+        grep -c active /proc/mdstat > mdadm_active2
+        ls /dev/disk/by-dname > ls_dname
+        """)]
+
+    def test_mdadm_output_files_exist(self):
+        self.output_files_exist(
+            ["fstab", "mdadm_status", "mdadm_active1", "mdadm_active2",
+             "ls_dname"])
+
+    def test_mdadm_status(self):
+        # ubuntu:<ID> is the name assigned to the md array
+        self.check_file_regex("mdadm_status", r"ubuntu:[0-9]*")
+        self.check_file_strippedline("mdadm_active1", self.active_mdadm)
+        self.check_file_strippedline("mdadm_active2", self.active_mdadm)
+
+
+class TestMdadmBcacheAbs(TestMdadmAbs):
+    conf_file = "examples/tests/raid5bcache.yaml"
+    disk_to_check = {'sda': 2,
+                     'md0': 0}
+
+    collect_scripts = TestMdadmAbs.collect_scripts + [textwrap.dedent("""
+        cd OUTPUT_COLLECT_D
+        bcache-super-show /dev/vda2 > bcache_super_vda2
+        ls /sys/fs/bcache > bcache_ls
+        cat /sys/block/bcache0/bcache/cache_mode > bcache_cache_mode
+        cat /proc/mounts > proc_mounts
+        cat /proc/partitions > proc_partitions
+        """)]
+    fstab_expected = {
+        '/dev/bcache0': '/',
+        '/dev/md0': '/srv/data',
+    }
+
+    def test_bcache_output_files_exist(self):
+        self.output_files_exist(["bcache_super_vda2", "bcache_ls",
+                                 "bcache_cache_mode"])
+
+    def test_bcache_status(self):
+        bcache_cset_uuid = None
+        with open(os.path.join(self.td.mnt, "bcache_super_vda2"), "r") as fp:
+            for line in fp.read().splitlines():
+                if line != "" and line.split()[0] == "cset.uuid":
+                    bcache_cset_uuid = line.split()[-1].rstrip()
+        self.assertIsNotNone(bcache_cset_uuid)
+        with open(os.path.join(self.td.mnt, "bcache_ls"), "r") as fp:
+            self.assertTrue(bcache_cset_uuid in fp.read().splitlines())
+
+    def test_bcache_cachemode(self):
+        self.check_file_regex("bcache_cache_mode", r"\[writeback\]")
+
+
+class WilyTestRaid5Bcache(TestMdadmBcacheAbs):
+    __test__ = True
+    release = "wily"
+
+
+class VividTestRaid5Bcache(TestMdadmBcacheAbs):
+    __test__ = True
+    release = "vivid"
+
+class TrustTestRaid5Bcache(TestMdadmBcacheAbs):
+    __test__ = True
+    release = "trusty"

=== modified file 'tools/launch'
--- tools/launch	2015-11-13 20:04:48 +0000
+++ tools/launch	2015-11-18 18:13:59 +0000
@@ -35,6 +35,8 @@
                         Note, qemu adds 5900 to port numbers. (:0 = port 5900)
            --serial-log F  : log to F (default 'serial.log')
       -v | --verbose    be more verbose
+           --no-install-deps  do not install insert '--install-deps'
+                              on curtin command invocations
 
    use of --kernel/--initrd will seed cloud-init via cmdline
    rather than the local datasource
@@ -99,6 +101,36 @@
 EOF
 }
 
+write_pstate_config() {
+    local pstate="$1" config1="$2" config2="$3"
+    cat > "$config1" <<EOF
+#cloud-config
+power_state:
+  mode: "$pstate"
+EOF
+    sed -e "s,PSTATE,$pstate," > "$config2" <<"EOF"
+#upstart-job
+# precise does not do cloud-config poweroff
+description "power-state for precise"
+start on stopped cloud-final
+console output
+task
+script
+  [ "$(lsb_release -sc)" = "precise" ] || exit 0
+  target="PSTATE"
+  msg="precise-powerstate: $target"
+  case "$target" in
+     on) exit 0;;
+     off|poweroff) shutdown -P now "$msg";;
+     reboot) shutdown -r now "$msg";;
+     *) echo "$msg : unknown target"; exit 1;;
+  esac
+  echo "$msg"
+  exit 0
+end script
+EOF
+}
+
 write_userdata() {
     local x
     cat <<EOF
@@ -219,7 +251,7 @@
     local initrd="" kernel="" uappend="" iargs="" disk_args=""
     local pubs="" disks="" pstate="null"
     local uefi=0
-    local netdevs=""
+    local netdevs="" install_deps="--install-deps"
     local video="-curses -vga std" serial_log="file:serial.log"
     # dowait: run xkvm with a '&' and then 'wait' on the pid.
     #  the reason to do this or not do this has to do with interactivity
@@ -246,6 +278,7 @@
             -d|--disk) disks[${#disks[@]}]="$next"; shift;;
             -h|--help) Usage ; exit 0;;
             -i|--initrd) initrd="$next"; shift;;
+               --no-install-deps) install_deps="";;
             -k|--kernel) kernel="$next"; shift;;
                --mem) mem="$next"; shift;;
                --power)
@@ -400,6 +433,12 @@
         addargs[${#addargs[@]}]="--add=config/${fpath##*/}:$fpath"
     done
 
+    if [ "${cmdargs[0]}" = "curtin" -a -n "$install_deps" ]; then
+        # if this is a 'curtin' command, then also insert --install-deps
+        debug 1 "adding --install-deps to command"
+        cmdargs=( curtin $install_deps "${cmdargs[@]:1}" )
+    fi
+
     PYTHONPATH="${top_d}${PYTHONPATH:+${PYTHONPATH}}" \
         "${top_d}/bin/curtin" pack "${addargs[@]}" -- \
         "${cmdargs[@]}" > "${TEMP_D}/install-cmd" ||
@@ -411,9 +450,9 @@
     local ccfiles=""
     ccfiles=( )
     if [ -n "${pstate}" ]; then
-        printf "#cloud-config\npower_state:\n  mode: %s\n" "$pstate" \
-            > "${TEMP_D}/pstate"
-        ccfiles[${#ccfiles[@]}]="${TEMP_D}/pstate"
+        write_pstate_config "$pstate" "${TEMP_D}/pstate.1" "${TEMP_D}/pstate.2"
+        ccfiles[${#ccfiles[@]}]="${TEMP_D}/pstate.1"
+        ccfiles[${#ccfiles[@]}]="${TEMP_D}/pstate.2"
     fi
 
     if tmp=$(find_apt_proxy); then

=== modified file 'tools/write-curtin'
--- tools/write-curtin	2015-03-09 16:35:00 +0000
+++ tools/write-curtin	2015-11-18 18:13:59 +0000
@@ -16,8 +16,7 @@
     parg = path
 
 ret = write_exe_wrapper(entrypoint="curtin.commands.main", path=parg,
-                        deps_check_entry="curtin.deps.check",
-                        deps_install_entry="curtin.deps.install")
+                        deps_check_entry="curtin.deps.check")
 
 if path == "-":
     sys.stdout.write(ret)