Merge lp:~raharper/curtin/trunk.implement_raid_preserve into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

block_meta: handle 'preserve' flag for raid devices

The raid handler did not respect the 'preserve' flag and
attempted to recreate the raid device instead of verifying
it.

Migrate mdadm usage into curtin.block.mdadm module.
Complete mdadm unittest for new module.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:316
https://server-team-jenkins.canonical.com/job/curtin-ci/23/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/21/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/23/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.implement_raid_preserve]

317. By Ryan Harper on 2015-12-07

from trunk

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:317
https://server-team-jenkins.canonical.com/job/curtin-ci/41/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/39/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/41/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

our block_meta has been what really *should* be curtin/block/__init__.py
there are a lot of functions there that are general purpose, not specifically tied to 'block_meta'.

just after having been looking through code on many different MP's recently, i've sen a lot of stuff in block_meta that i'd like in block/__init__.py.

[Ryan Harper (raharper)]

On Mon, Dec 7, 2015 at 10:27 AM, Scott Moser <email address hidden> wrote:

> our block_meta has been what really *should* be curtin/block/__init__.py
> there are a lot of functions there that are general purpose, not
> specifically tied to 'block_meta'.
>
> just after having been looking through code on many different MP's
> recently, i've sen a lot of stuff in block_meta that i'd like in
> block/__init__.py.
>

That's good to know. I don't mind moving them out there.

>
>
> Diff comments:
>
> > === modified file 'curtin/commands/block_meta.py'
> > --- curtin/commands/block_meta.py 2015-11-18 18:12:54 +0000
> > +++ curtin/commands/block_meta.py 2015-12-07 15:22:18 +0000
> > @@ -438,6 +438,75 @@
> > return volume_path
> >
> >
> > +def check_raid_array(mdname, raidlevel, devices=[], spares=[]):
> > + LOG.debug('Checking mdadm array: '
> > + 'name={} raidlevel={} devices={} spares={}'.format(
> > + mdname, raidlevel, devices, spares))
> > +
> > + # query the target raid device
> > + md_devname = os.path.join('/dev', mdname)
>
> beware os.path.join("/dev/", "/tmp/foo") returns /tmp/foo.
>

I think we're more concerned about:

 os.path.join("/dev", "/tmp/foo")
'/tmp/foo'

> I'd like a general get_full_device_path_for_device_name(devname="sda")
> that woudl return /dev/sda or what not.
>

I think most of it is here (but expects the volume id)

def get_path_to_storage_volume()

Instead you want it to take what? kernel devnames? /dev/XXXX, or XXX?
Is this just a smarter os.path.join for devices?

>
> > + (out, _err) = util.subp(["mdadm", "--query", "--detail", "--export",
>
> this next block of code should be in a function. mdadm_query_devname()
>

OK

>
> > + md_devname], capture=True)
> > + if _err:
> > + # this is not fatal as we may need to assemble the array
> > + LOG.warn("raid device '%s' does not exist" % md_devname)
> > + return False
> > +
> > + # Convert mdadm --query --detail --export key=value into dictionary
> > + md_query_data = {k: v for k, v in (x.split('=')
>
> I'm guessing you want to use shlex here.
> see curtin/block/__init__.py's blkid() which returns a dictionary and
> parses output using shlex.
>

Hrm, do you want me to make a general:

def subp_to_dict(cmd)

?

And have blkid use that? the K=V logic is currently embedded in there.

>
> > + for x in out.strip().split('\n'))}
> > +
> > + # confirm we have /dev/{mdname} by following the udev symlink
> > + mduuid_path = ('/dev/disk/by-id/md-uuid-' +
> > + '{}'.format(md_query_data['MD_UUID']))
> > + mdquery_devname = os.path.realpath(mduuid_path)
> > + if md_devname != mdquery_devname:
>
> this is a good check thank.s
>

NP, we should probably add that to the vmtests raid check as well.

>
> > + raise ValueError("Couldn't find correct raid device."
>
> need a ' ' after the '.' there or your value eror shows
> "device.MD_UUID={}".
>
>
OK

> > + "MD_UUID={} points to {} but expected {}" % (
> > + md_query_data['MD_UUID'],
> > + ...

[lp:~raharper/curtin/trunk.implement_raid_preserve]

318. By Ryan Harper on 2015-12-07

from trunk

319. By Ryan Harper on 2015-12-09

Fix spacing in raid check exception message.

320. By Ryan Harper on 2015-12-09

from trunk

321. By Ryan Harper on 2015-12-17

Create block.mdadm module

322. By Ryan Harper on 2015-12-17

Add mdadm unittests, apply some fixes and additional checks to mdadm module.

323. By Ryan Harper on 2015-12-17

mdadm: code-motion, relocating out of mdadm_ section.

324. By Ryan Harper on 2015-12-17

unittests: add mdadm_stop, mdadm_remove tests

325. By Ryan Harper on 2015-12-17

unittests: add mdadm_detail_scan

326. By Ryan Harper on 2015-12-17

unittests: add mdadm_query_detail

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:326
https://server-team-jenkins.canonical.com/job/curtin-ci/55/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/52/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/55/rebuild

review: Needs Fixing (continuous-integration)

[lp:~raharper/curtin/trunk.implement_raid_preserve]

327. By Ryan Harper on 2015-12-17

unittests: mdadm helpers

328. By Ryan Harper on 2015-12-18

unittests: mdadm md_helpers

329. By Ryan Harper on 2015-12-18

mdadm: use util.load_file for easier mocking

330. By Ryan Harper on 2015-12-18

unittests: mdadm md_check_raidlevel

331. By Ryan Harper on 2015-12-18

unittests: mdadm md_check_array_state

332. By Ryan Harper on 2015-12-18

fix lint/pep8

333. By Ryan Harper on 2015-12-18

unittests: mdadm md_check_{uuid,devices,spares}

334. By Ryan Harper on 2015-12-18

unittest: fix pep/lint

335. By Ryan Harper on 2015-12-18

unittests: mdadm finish md_helpers

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:335
https://server-team-jenkins.canonical.com/job/curtin-ci/71/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/68/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/71/rebuild

review: Needs Fixing (continuous-integration)

[lp:~raharper/curtin/trunk.implement_raid_preserve]

336. By Ryan Harper on 2015-12-18

from trunk

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:336
https://server-team-jenkins.canonical.com/job/curtin-ci/73/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/70/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/73/rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

this alrgely looks good.
I'm ok if you want to push on it and get it in, given the good tests that youv'e got in place.
Just make sure we're passing vmtest and unit tests first.

i wont be able to give more thourough review till the new year.

[Ryan Harper (raharper)]

OK, I won't push until it passes vmtests, it already passes make check.

On Fri, Dec 18, 2015 at 4:23 PM, Scott Moser <email address hidden> wrote:

> this alrgely looks good.
> I'm ok if you want to push on it and get it in, given the good tests that
> youv'e got in place.
> Just make sure we're passing vmtest and unit tests first.
>
> i wont be able to give more thourough review till the new year.
>
>
> --
>
> https://code.launchpad.net/~raharper/curtin/trunk.implement_raid_preserve/+merge/279603
> You are the owner of lp:~raharper/curtin/trunk.implement_raid_preserve.
>

[lp:~raharper/curtin/trunk.implement_raid_preserve]

337. By Ryan Harper on 2015-12-19

Don't expect md_devname to exist when creating raid devices.

338. By Ryan Harper on 2015-12-19

reformat to fix pep on newer systems

339. By Ryan Harper on 2015-12-19

Allow non-zero return codes for mdadm_assemble

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:339
https://server-team-jenkins.canonical.com/job/curtin-ci/76/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/73/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/76/rebuild

review: Needs Fixing (continuous-integration)

[lp:~raharper/curtin/trunk.implement_raid_preserve]

340. By Ryan Harper on 2015-12-19

fix assert_has_calls to use list

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:340
https://server-team-jenkins.canonical.com/job/curtin-ci/77/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/74/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/77/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.implement_raid_preserve]

341. By Ryan Harper on 2016-01-04

Rename block.dev_long -> block.dev_path

Replacing dev_long with dev_path to indicate that we're returning a device
path.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:341
https://server-team-jenkins.canonical.com/job/curtin-ci/85/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/82/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/85/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Locally I ran make check and make vmtest succesfully.
I merged into trunk and pushed.
Fix-committed in revno 330.

Thanks Ryan.

review: Approve

=== modified file 'curtin/block/__init__.py'
--- curtin/block/__init__.py	2015-12-07 20:22:00 +0000
+++ curtin/block/__init__.py	2016-01-04 16:25:53 +0000
@@ -41,6 +41,23 @@
     return False
 
 
+def dev_short(devname):
+    if os.path.sep in devname:
+        return os.path.basename(devname)
+    return devname
+
+
+def dev_path(devname):
+    if devname.startswith('/dev/'):
+        return devname
+    else:
+        return '/dev/' + devname
+
+
+def sys_block_path(devname):
+    return '/sys/class/block/' + dev_short(devname)
+
+
 def _lsblock_pairs_to_dict(lines):
     ret = {}
     for line in lines.splitlines():
@@ -435,4 +452,5 @@
             does not exist" % (path, serial))
     return path
 
+
 # vi: ts=4 expandtab syntax=python

=== added file 'curtin/block/mdadm.py'
--- curtin/block/mdadm.py	1970-01-01 00:00:00 +0000
+++ curtin/block/mdadm.py	2016-01-04 16:25:53 +0000
@@ -0,0 +1,630 @@
+#   Copyright (C) 2015 Canonical Ltd.
+#
+#   Author: Ryan Harper <[e-mail]>
+#
+#   Curtin is free software: you can redistribute it and/or modify it under
+#   the terms of the GNU Affero General Public License as published by the
+#   Free Software Foundation, either version 3 of the License, or (at your
+#   option) any later version.
+#
+#   Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
+#   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+#   FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for
+#   more details.
+#
+#   You should have received a copy of the GNU Affero General Public License
+#   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
+
+
+# This module wraps calls to the mdadm utility for examing Linux SoftRAID
+# virtual devices.  Functions prefixed with 'mdadm_' involve executing
+# the 'mdadm' command in a subprocess.  The remaining functions handle
+# manipulation of the mdadm output.
+
+
+import os
+import re
+import shlex
+from subprocess import CalledProcessError
+
+from curtin.block import (dev_short, dev_path, is_valid_device, sys_block_path)
+from curtin import util
+from curtin.log import LOG
+
+NOSPARE_RAID_LEVELS = [
+    'linear', 'raid0', '0', 0,
+]
+
+SPARE_RAID_LEVELS = [
+    'raid1', 'stripe', 'mirror', '1', 1,
+    'raid4', '4', 4,
+    'raid5', '5', 5,
+    'raid6', '6', 6,
+    'raid10', '10', 10,
+]
+
+VALID_RAID_LEVELS = NOSPARE_RAID_LEVELS + SPARE_RAID_LEVELS
+
+#  https://www.kernel.org/doc/Documentation/md.txt
+'''
+     clear
+         No devices, no size, no level
+         Writing is equivalent to STOP_ARRAY ioctl
+     inactive
+         May have some settings, but array is not active
+            all IO results in error
+         When written, doesn't tear down array, but just stops it
+     suspended (not supported yet)
+         All IO requests will block. The array can be reconfigured.
+         Writing this, if accepted, will block until array is quiessent
+     readonly
+         no resync can happen.  no superblocks get written.
+         write requests fail
+     read-auto
+         like readonly, but behaves like 'clean' on a write request.
+
+     clean - no pending writes, but otherwise active.
+         When written to inactive array, starts without resync
+         If a write request arrives then
+           if metadata is known, mark 'dirty' and switch to 'active'.
+           if not known, block and switch to write-pending
+         If written to an active array that has pending writes, then fails.
+     active
+         fully active: IO and resync can be happening.
+         When written to inactive array, starts with resync
+
+     write-pending
+         clean, but writes are blocked waiting for 'active' to be written.
+
+     active-idle
+       like active, but no writes have been seen for a while (safe_mode_delay).
+'''
+
+ERROR_RAID_STATES = [
+    'clear',
+    'inactive',
+    'suspended',
+]
+
+READONLY_RAID_STATES = [
+    'readonly',
+]
+
+READWRITE_RAID_STATES = [
+    'read-auto',
+    'clean',
+    'active',
+    'active-idle',
+    'write-pending',
+]
+
+VALID_RAID_ARRAY_STATES = (
+    ERROR_RAID_STATES +
+    READONLY_RAID_STATES +
+    READWRITE_RAID_STATES
+)
+
+# need a on-import check of version and set the value for later reference
+''' mdadm version < 3.3 doesn't include enough info when using --export
+    and we must use --detail and parse out information.  This method
+    checks the mdadm version and will return True if we can use --export
+    for key=value list with enough info, false if version is less than
+'''
+MDADM_USE_EXPORT = util.lsb_release()['codename'] not in ['precise', 'trusty']
+
+#
+# mdadm executors
+#
+
+
+def mdadm_assemble(md_devname=None, devices=[], spares=[], scan=False):
+    # md_devname is a /dev/XXXX
+    # devices is non-empty list of /dev/xxx
+    # if spares is non-empt list append of /dev/xxx
+    cmd = ["mdadm", "--assemble"]
+    if scan:
+        cmd += ['--scan']
+    else:
+        valid_mdname(md_devname)
+        cmd += [dev_path(md_devname), "--run"] + devices
+        if spares:
+            cmd += spares
+
+    util.subp(cmd, capture=True, rcs=[0, 1, 2])
+    util.subp(["udevadm", "settle"])
+
+
+def mdadm_create(md_devname, raidlevel, devices, spares=None, md_name=""):
+    LOG.debug('mdadm_create: ' +
+              'md_name=%s raidlevel=%s ' % (md_devname, raidlevel) +
+              ' devices=%s spares=%s name=%s' % (devices, spares, md_name))
+
+    if raidlevel not in VALID_RAID_LEVELS:
+        raise ValueError('Invalid raidlevel: ' + str(raidlevel))
+
+    min_devices = md_minimum_devices(raidlevel)
+    if len(devices) < min_devices:
+        err = 'Not enough devices for raidlevel: ' + str(raidlevel)
+        err += ' minimum devices needed: ' + str(min_devices)
+        raise ValueError(err)
+
+    if spares and raidlevel not in SPARE_RAID_LEVELS:
+        err = ('Raidlevel does not support spare devices: ' + str(raidlevel))
+        raise ValueError(err)
+
+    cmd = ["mdadm", "--create", dev_path(md_devname), "--run",
+           "--level=%s" % raidlevel, "--raid-devices=%s" % len(devices)]
+    if md_name:
+        cmd.append("--name=%s" % md_name)
+
+    for device in devices:
+        # Zero out device superblock just in case device has been used for raid
+        # before, as this will cause many issues
+        util.subp(["mdadm", "--zero-superblock", device], capture=True)
+        cmd.append(device)
+
+    if spares:
+        cmd.append("--spare-devices=%s" % len(spares))
+        for device in spares:
+            util.subp(["mdadm", "--zero-superblock", device], capture=True)
+            cmd.append(device)
+
+    # Create the raid device
+    util.subp(["udevadm", "settle"])
+    util.subp(["udevadm", "control", "--stop-exec-queue"])
+    util.subp(cmd, capture=True)
+    util.subp(["udevadm", "control", "--start-exec-queue"])
+    util.subp(["udevadm", "settle", "--exit-if-exists=%s" % md_devname])
+
+
+def mdadm_examine(devpath, export=MDADM_USE_EXPORT):
+    ''' exectute mdadm --examine, and optionally
+        append --export.
+        Parse and return dict of key=val from output'''
+    cmd = ["mdadm", "--examine"]
+    if export:
+        cmd.extend(["--export"])
+
+    cmd.extend([devpath])
+    try:
+        (out, _err) = util.subp(cmd, capture=True)
+    except CalledProcessError:
+        LOG.exception('Error: not a valid md device: ' + devpath)
+        return {}
+
+    if export:
+        data = __mdadm_export_to_dict(out)
+    else:
+        data = __upgrade_detail_dict(__mdadm_detail_to_dict(out))
+
+    return data
+
+
+def mdadm_stop(devpath):
+    if not devpath:
+        raise ValueError('mdadm_stop: missing parameter devpath')
+
+    LOG.info("mdadm stopping: %s" % devpath)
+    util.subp(["mdadm", "--stop", devpath], rcs=[0, 1], capture=True)
+
+
+def mdadm_remove(devpath):
+    if not devpath:
+        raise ValueError('mdadm_remove: missing parameter devpath')
+
+    LOG.info("mdadm removing: %s" % devpath)
+    util.subp(["mdadm", "--remove", devpath], rcs=[0, 1], capture=True)
+
+
+def mdadm_query_detail(md_devname, export=MDADM_USE_EXPORT):
+    valid_mdname(md_devname)
+
+    cmd = ["mdadm", "--query", "--detail"]
+    if export:
+        cmd.extend(["--export"])
+    cmd.extend([md_devname])
+    (out, _err) = util.subp(cmd, capture=True)
+
+    if export:
+        data = __mdadm_export_to_dict(out)
+    else:
+        data = __upgrade_detail_dict(__mdadm_detail_to_dict(out))
+
+    return data
+
+
+def mdadm_detail_scan():
+    (out, _err) = util.subp(["mdadm", "--detail", "--scan"], capture=True)
+    if not _err:
+        return out
+
+
+# ------------------------------ #
+def valid_mdname(md_devname):
+    if md_devname is None:
+        raise ValueError('Parameter: md_devname is None')
+        return False
+
+    if not is_valid_device(dev_path(md_devname)):
+        raise ValueError('Specified md device does not exist: ' +
+                         dev_path(md_devname))
+        return False
+
+    return True
+
+
+def md_sysfs_attr(md_devname, attrname):
+    if not valid_mdname(md_devname):
+        raise ValueError('Invalid md devicename')
+
+    attrdata = ''
+    #  /sys/class/block/<md_short>/md
+    sysmd = sys_block_path(dev_short(md_devname)) + "/md"
+
+    #  /sys/class/block/<md_short>/md/attrname
+    sysfs_attr_path = os.path.join(sysmd, attrname)
+    if not os.path.isfile(sysfs_attr_path):
+        with open(sysfs_attr_path) as fp:
+            attrdata = fp.read().strip()
+
+    return attrdata
+
+
+def md_raidlevel_short(raidlevel):
+    if isinstance(raidlevel, int) or raidlevel in ['linear', 'stripe']:
+        return raidlevel
+
+    return int(raidlevel.replace('raid', ''))
+
+
+def md_minimum_devices(raidlevel):
+    ''' return the minimum number of devices for a given raid level '''
+    rl = md_raidlevel_short(raidlevel)
+    if rl in [0, 1, 'linear', 'stripe']:
+        return 2
+    if rl in [5]:
+        return 3
+    if rl in [6, 10]:
+        return 4
+
+    return -1
+
+
+def __md_check_array_state(md_devname, mode='READWRITE'):
+    modes = {
+        'READWRITE': READWRITE_RAID_STATES,
+        'READONLY': READONLY_RAID_STATES,
+        'ERROR': ERROR_RAID_STATES,
+    }
+    if mode not in modes:
+        raise ValueError('Invalid Array State mode: ' + mode)
+
+    array_state = md_sysfs_attr(md_devname, 'array_state')
+    if array_state in modes[mode]:
+        return True
+
+    return False
+
+
+def md_check_array_state_rw(md_devname):
+    return __md_check_array_state(md_devname, mode='READWRITE')
+
+
+def md_check_array_state_ro(md_devname):
+    return __md_check_array_state(md_devname, mode='READONLY')
+
+
+def md_check_array_state_error(md_devname):
+    return __md_check_array_state(md_devname, mode='ERROR')
+
+
+def __mdadm_export_to_dict(output):
+    ''' convert Key=Value text output into dictionary '''
+    return dict(tok.split('=', 1) for tok in shlex.split(output))
+
+
+def __mdadm_detail_to_dict(input):
+    ''' Convert mdadm --detail output to dictionary
+
+    /dev/vde:
+              Magic : a92b4efc
+            Version : 1.2
+        Feature Map : 0x0
+         Array UUID : 93a73e10:427f280b:b7076c02:204b8f7a
+               Name : wily-foobar:0  (local to host wily-foobar)
+      Creation Time : Sat Dec 12 16:06:05 2015
+         Raid Level : raid1
+       Raid Devices : 2
+
+     Avail Dev Size : 20955136 (9.99 GiB 10.73 GB)
+      Used Dev Size : 20955136 (9.99 GiB 10.73 GB)
+         Array Size : 10477568 (9.99 GiB 10.73 GB)
+        Data Offset : 16384 sectors
+       Super Offset : 8 sectors
+       Unused Space : before=16296 sectors, after=0 sectors
+              State : clean
+        Device UUID : 8fcd62e6:991acc6e:6cb71ee3:7c956919
+
+        Update Time : Sat Dec 12 16:09:09 2015
+      Bad Block Log : 512 entries available at offset 72 sectors
+           Checksum : 65b57c2e - correct
+             Events : 17
+
+
+       Device Role : spare
+       Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
+    '''
+    data = {}
+
+    device = re.findall('^(\/dev\/[a-zA-Z0-9-\._]+)', input)
+    if len(device) == 1:
+        data.update({'device': device[0]})
+    else:
+        raise ValueError('Failed to determine device in input')
+
+    #  FIXME: probably could do a better regex to match the LHS which
+    #         has one, two or three words
+    for f in re.findall('(\w+|\w+\ \w+|\w+\ \w+\ \w+)' +
+                        '\ \:\ ([a-zA-Z0-9\-\.,: \(\)=\']+)',
+                        input, re.MULTILINE):
+        key = f[0].replace(' ', '_').lower()
+        val = f[1]
+        if key in data:
+            raise ValueError('Duplicate key in mdadm regex parsing: ' + key)
+        data.update({key: val})
+
+    return data
+
+
+def md_device_key_role(devname):
+    if not devname:
+        raise ValueError('Missing parameter devname')
+    return 'MD_DEVICE_' + dev_short(devname) + '_ROLE'
+
+
+def md_device_key_dev(devname):
+    if not devname:
+        raise ValueError('Missing parameter devname')
+    return 'MD_DEVICE_' + dev_short(devname) + '_DEV'
+
+
+def __upgrade_detail_dict(detail):
+    ''' This method attempts to convert mdadm --detail output into
+        a KEY=VALUE output the same as mdadm --detail --export from mdadm v3.3
+    '''
+    # if the input already has MD_UUID, it's already been converted
+    if 'MD_UUID' in detail:
+        return detail
+
+    md_detail = {
+        'MD_LEVEL': detail['raid_level'],
+        'MD_DEVICES': detail['raid_devices'],
+        'MD_METADATA': detail['version'],
+        'MD_NAME': detail['name'].split()[0],
+    }
+
+    # exmaine has ARRAY UUID
+    if 'array_uuid' in detail:
+        md_detail.update({'MD_UUID': detail['array_uuid']})
+    # query,detail has UUID
+    elif 'uuid' in detail:
+        md_detail.update({'MD_UUID': detail['uuid']})
+
+    device = detail['device']
+
+    #  MD_DEVICE_vdc1_DEV=/dev/vdc1
+    md_detail.update({md_device_key_dev(device): device})
+
+    if 'device_role' in detail:
+        role = detail['device_role']
+        if role != 'spare':
+            # device_role = Active device 1
+            role = role.split()[-1]
+
+        # MD_DEVICE_vdc1_ROLE=spare
+        md_detail.update({md_device_key_role(device): role})
+
+    return md_detail
+
+
+def md_read_run_mdadm_map():
+    '''
+        md1 1.2 59beb40f:4c202f67:088e702b:efdf577a /dev/md1
+        md0 0.90 077e6a9e:edf92012:e2a6e712:b193f786 /dev/md0
+
+        return
+        # md_shortname = (metaversion, md_uuid, md_devpath)
+        data = {
+            'md1': (1.2, 59beb40f:4c202f67:088e702b:efdf577a, /dev/md1)
+            'md0': (0.90, 077e6a9e:edf92012:e2a6e712:b193f786, /dev/md0)
+    '''
+
+    mdadm_map = {}
+    run_mdadm_map = '/run/mdadm/map'
+    if os.path.exists(run_mdadm_map):
+        with open(run_mdadm_map, 'r') as fp:
+            data = fp.read().strip()
+        for entry in data.split('\n'):
+            (key, meta, md_uuid, dev) = entry.split()
+            mdadm_map.update({key: (meta, md_uuid, dev)})
+
+    return mdadm_map
+
+
+def md_get_spares_list(devpath):
+    sysfs_md = sys_block_path(devpath) + '/md'
+
+    if not os.path.exists(sysfs_md):
+        raise ValueError('Cannot find md sysfs directory: ' +
+                         sysfs_md)
+
+    spares = [dev_path(dev[4:])
+              for dev in os.listdir(sysfs_md)
+              if (dev.startswith('dev-') and
+                  util.load_file(os.path.join(sysfs_md,
+                                              dev,
+                                              'state')).strip() == 'spare')]
+
+    return spares
+
+
+def md_get_devices_list(devpath):
+    sysfs_md = sys_block_path(devpath) + '/md'
+    if not os.path.exists(sysfs_md):
+        raise ValueError('Cannot find md sysfs directory: ' +
+                         sysfs_md)
+    devices = [dev_path(dev[4:])
+               for dev in os.listdir(sysfs_md)
+               if (dev.startswith('dev-') and
+                   util.load_file(os.path.join(sysfs_md,
+                                               dev,
+                                               'state')).strip() != 'spare')]
+    return devices
+
+
+def md_check_array_uuid(md_devname, md_uuid):
+    # confirm we have /dev/{mdname} by following the udev symlink
+    mduuid_path = ('/dev/disk/by-id/md-uuid-' + md_uuid)
+    mdlink_devname = dev_path(os.path.realpath(mduuid_path))
+    if md_devname != mdlink_devname:
+        err = ('Mismatch between devname and md-uuid symlink: ' +
+               '%s -> %s != %s' % (mduuid_path, mdlink_devname, md_devname))
+        raise ValueError(err)
+
+    return True
+
+
+def md_get_uuid(md_devname):
+    if not valid_mdname(md_devname):
+        raise ValueError('Invalid md devicename')
+
+    md_query = mdadm_query_detail(md_devname)
+    return md_query.get('MD_UUID', None)
+
+
+def _compare_devlist(expected, found):
+    LOG.debug('comparing device lists: '
+              'expected: {} found: {}'.format(expected, found))
+    expected = set(expected)
+    found = set(found)
+    if expected != found:
+        missing = expected.difference(found)
+        extra = found.difference(expected)
+        raise ValueError("RAID array device list does not match."
+                         " Missing: {} Extra: {}".format(missing, extra))
+
+
+def md_check_raidlevel(raidlevel):
+    # Validate raidlevel against what curtin supports configuring
+    if raidlevel not in VALID_RAID_LEVELS:
+        err = ('Invalid raidlevel: ' + raidlevel +
+               ' Must be one of: ' + str(VALID_RAID_LEVELS))
+        raise ValueError(err)
+    return True
+
+
+def md_block_until_in_sync(md_devname):
+    '''
+    sync_completed
+    This shows the number of sectors that have been completed of
+    whatever the current sync_action is, followed by the number of
+    sectors in total that could need to be processed.  The two
+    numbers are separated by a '/'  thus effectively showing one
+    value, a fraction of the process that is complete.
+    A 'select' on this attribute will return when resync completes,
+    when it reaches the current sync_max (below) and possibly at
+    other times.
+    '''
+    # FIXME: use selectors to block on: /sys/class/block/mdX/md/sync_completed
+    pass
+
+
+def md_check_array_state(md_devname):
+    # check array state
+
+    writable = md_check_array_state_rw(md_devname)
+    degraded = int(md_sysfs_attr(md_devname, 'degraded'))
+    sync_action = md_sysfs_attr(md_devname, 'sync_action')
+
+    if not writable:
+        raise ValueError('Array not in writable state: ' + md_devname)
+    if degraded > 0:
+        raise ValueError('Array in degraded state: ' + md_devname)
+    if sync_action != "idle":
+        raise ValueError('Array syncing, not idle state: ' + md_devname)
+
+    return True
+
+
+def md_check_uuid(md_devname):
+    md_uuid = md_get_uuid(md_devname)
+    if not md_uuid:
+        raise ValueError('Failed to get md UUID from device: ' + md_devname)
+    return md_check_array_uuid(md_devname, md_uuid)
+
+
+def md_check_devices(md_devname, devices):
+    if not devices or len(devices) == 0:
+        raise ValueError('Cannot verify raid array with empty device list')
+
+    # collect and compare raid devices based on md name versus
+    # expected device list.
+    #
+    # NB: In some cases, a device might report as a spare until
+    #     md has finished syncing it into the array.  Currently
+    #     we fail the check since the specified raid device is not
+    #     yet in its proper role.  Callers can check mdadm_sync_action
+    #     state to see if the array is currently recovering, which would
+    #     explain the failure.  Also  mdadm_degraded will indicate if the
+    #     raid is currently degraded or not, which would also explain the
+    #     failure.
+    md_raid_devices = md_get_devices_list(md_devname)
+    LOG.debug('md_check_devices: md_raid_devs: ' + str(md_raid_devices))
+    _compare_devlist(devices, md_raid_devices)
+
+
+def md_check_spares(md_devname, spares):
+    # collect and compare spare devices based on md name versus
+    # expected device list.
+    md_raid_spares = md_get_spares_list(md_devname)
+    _compare_devlist(spares, md_raid_spares)
+
+
+def md_check_array_membership(md_devname, devices):
+    # validate that all devices are members of the correct array
+    md_uuid = md_get_uuid(md_devname)
+    for device in devices:
+        dev_examine = mdadm_examine(device, export=False)
+        if 'MD_UUID' not in dev_examine:
+            raise ValueError('Device is not part of an array: ' + device)
+        dev_uuid = dev_examine['MD_UUID']
+        if dev_uuid != md_uuid:
+            err = "Device {} is not part of {} array. ".format(device,
+                                                               md_devname)
+            err += "MD_UUID mismatch: device:{} != array:{}".format(dev_uuid,
+                                                                    md_uuid)
+            raise ValueError(err)
+
+
+def md_check(md_devname, raidlevel, devices=[], spares=[]):
+    ''' Check passed in variables from storage configuration against
+        the system we're running upon.
+    '''
+    LOG.debug('RAID validation: ' +
+              'name={} raidlevel={} devices={} spares={}'.format(md_devname,
+                                                                 raidlevel,
+                                                                 devices,
+                                                                 spares))
+
+    md_check_array_state(md_devname)
+    md_check_raidlevel(raidlevel)
+    md_check_uuid(md_devname)
+    md_check_devices(md_devname, devices)
+    md_check_spares(md_devname, spares)
+    md_check_array_membership(md_devname, devices + spares)
+
+    LOG.debug('RAID array OK: ' + md_devname)
+    return True
+
+
+# vi: ts=4 expandtab syntax=python

=== modified file 'curtin/commands/block_meta.py'
--- curtin/commands/block_meta.py	2015-12-08 01:23:09 +0000
+++ curtin/commands/block_meta.py	2016-01-04 16:25:53 +0000
@@ -16,9 +16,8 @@
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
 from collections import OrderedDict
-from curtin import block
-from curtin import config
-from curtin import util
+from curtin import (block, config, util)
+from curtin.block import mdadm
 from curtin.log import LOG
 
 from . import populate_one_subcmd
@@ -256,9 +255,8 @@
         block_dev = os.path.join("/dev/", os.path.split(sys_block_path)[-1])
         # if these fail its okay, the array might not be assembled and thats
         # fine
-        LOG.info("stopping: %s" % block_dev)
-        util.subp(["mdadm", "--stop", block_dev], rcs=[0, 1])
-        util.subp(["mdadm", "--remove", block_dev], rcs=[0, 1])
+        mdadm.mdadm_stop(block_dev)
+        mdadm.mdadm_remove(block_dev)
 
     elif os.path.exists(os.path.join(sys_block_path, "dm")):
         # Shut down any volgroups
@@ -341,12 +339,8 @@
             "-part%s" % determine_partition_number(volume, storage_config)
         rule.append(compose_udev_equality('ENV{ID_PART_ENTRY_UUID}', ptuuid))
     elif vol.get('type') == "raid":
-        (out, _err) = util.subp(["mdadm", "--detail", "--export", path],
-                                capture=True)
-        for line in out.splitlines():
-            if "MD_UUID" in line:
-                md_uuid = line.split('=')[-1]
-                break
+        md_data = mdadm.mdadm_query_detail(path)
+        md_uuid = md_data.get('MD_UUID')
         rule.append(compose_udev_equality("ENV{MD_UUID}", md_uuid))
     elif vol.get('type') == "bcache":
         rule.append(compose_udev_equality("ENV{DEVNAME}", path))
@@ -471,7 +465,7 @@
     # Wipe the disk
     if info.get('wipe') and info.get('wipe') != "none":
         # The disk has a lable, clear all partitions
-        util.subp(["mdadm", "--assemble", "--scan"], rcs=[0, 1, 2])
+        mdadm.mdadm_assemble(scan=True)
         disk_kname = os.path.split(disk)[-1]
         syspath_partitions = list(
             os.path.split(prt)[0] for prt in
@@ -914,39 +908,33 @@
     device_paths = list(get_path_to_storage_volume(dev, storage_config) for
                         dev in devices)
 
+    spare_device_paths = []
     if spare_devices:
         spare_device_paths = list(get_path_to_storage_volume(dev,
                                   storage_config) for dev in spare_devices)
 
-    mdnameparm = ""
-    mdname = info.get('mdname')
-    if mdname:
-        mdnameparm = "--name=%s" % info.get('mdname')
-
-    cmd = ["mdadm", "--create", "/dev/%s" % info.get('name'), "--run",
-           "--level=%s" % raidlevel, "--raid-devices=%s" % len(device_paths),
-           mdnameparm]
-
-    for device in device_paths:
-        # Zero out device superblock just in case device has been used for raid
-        # before, as this will cause many issues
-        util.subp(["mdadm", "--zero-superblock", device])
-
-        cmd.append(device)
-
-    if spare_devices:
-        cmd.append("--spare-devices=%s" % len(spare_device_paths))
-        for device in spare_device_paths:
-            util.subp(["mdadm", "--zero-superblock", device])
-
-            cmd.append(device)
-
-    # Create the raid device
-    util.subp(["udevadm", "settle"])
-    util.subp(["udevadm", "control", "--stop-exec-queue"])
-    util.subp(" ".join(cmd), shell=True)
-    util.subp(["udevadm", "control", "--start-exec-queue"])
-    util.subp(["udevadm", "settle"])
+    # Handle preserve flag
+    if info.get('preserve'):
+        # check if the array is already up, if not try to assemble
+        if not block.md_check(info.get('name'), raidlevel,
+                              device_paths, spare_device_paths):
+            LOG.info("assembling preserved raid for "
+                     "{}".format(info.get('name')))
+
+            mdadm.mdadm_assemble(info.get('name'),
+                                 device_paths, spare_device_paths)
+
+            # try again after attempting to assemble
+            if not mdadm.md_check(info.get('name'), raidlevel,
+                                  devices, spare_device_paths):
+                raise ValueError("Unable to confirm preserved raid array: "
+                                 " {}".format(info.get('name')))
+        # raid is all OK
+        return
+
+    mdadm.mdadm_create(info.get('name'), raidlevel,
+                       device_paths, spare_device_paths,
+                       info.get('mdname', ''))
 
     # Make dname rule for this dev
     make_dname(info.get('id'), storage_config)
@@ -958,9 +946,9 @@
     if state['fstab']:
         mdadm_location = os.path.join(os.path.split(state['fstab'])[0],
                                       "mdadm.conf")
-        (out, _err) = util.subp(["mdadm", "--detail", "--scan"], capture=True)
+        mdadm_scan_data = mdadm.mdadm_detail_scan()
         with open(mdadm_location, "w") as fp:
-            fp.write(out)
+            fp.write(mdadm_scan_data)
     else:
         LOG.info("fstab configuration is not present in the environment, so \
             cannot locate an appropriate directory to write mdadm.conf in, \

=== added file 'tests/unittests/test_block_mdadm.py'
--- tests/unittests/test_block_mdadm.py	1970-01-01 00:00:00 +0000
+++ tests/unittests/test_block_mdadm.py	2016-01-04 16:25:53 +0000
@@ -0,0 +1,891 @@
+from unittest import TestCase
+from mock import call, patch
+from curtin.block import dev_short
+from curtin.block import mdadm
+import os
+import subprocess
+
+from sys import version_info
+if version_info.major == 2:
+    import __builtin__ as builtins
+else:
+    import builtins
+
+
+class MdadmTestBase(TestCase):
+    def setUp(self):
+        super(MdadmTestBase, self).setUp()
+
+    def add_patch(self, target, attr):
+        """Patches specified target object and sets it as attr on test
+        instance also schedules cleanup"""
+        m = patch(target, autospec=True)
+        p = m.start()
+        self.addCleanup(m.stop)
+        setattr(self, attr, p)
+
+
+class TestBlockMdadmAssemble(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmAssemble, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        # Common mock settings
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'precise'}
+        self.mock_util.subp.side_effect = [
+            ("", ""),  # mdadm assemble
+            ("", ""),  # udevadm settle
+        ]
+
+    def test_mdadm_assemble_scan(self):
+        mdadm.mdadm_assemble(scan=True)
+        expected_calls = [
+            call(["mdadm", "--assemble", "--scan"], capture=True,
+                 rcs=[0, 1, 2]),
+            call(["udevadm", "settle"]),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+    def test_mdadm_assemble_md_devname(self):
+        md_devname = "/dev/md0"
+        mdadm.mdadm_assemble(md_devname=md_devname)
+
+        expected_calls = [
+            call(["mdadm", "--assemble", md_devname, "--run"], capture=True,
+                 rcs=[0, 1, 2]),
+            call(["udevadm", "settle"]),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+    def test_mdadm_assemble_md_devname_short(self):
+        md_devname = "md0"
+        mdadm.mdadm_assemble(md_devname=md_devname)
+
+        expected_calls = [
+            call(["mdadm", "--assemble", "/dev/md0", "--run"], capture=True,
+                 rcs=[0, 1, 2]),
+            call(["udevadm", "settle"]),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+    def test_mdadm_assemble_md_devname_none(self):
+        with self.assertRaises(ValueError):
+            md_devname = None
+            mdadm.mdadm_assemble(md_devname=md_devname)
+
+    def test_mdadm_assemble_md_devname_devices(self):
+        md_devname = "/dev/md0"
+        devices = ["/dev/vdc1", "/dev/vdd1"]
+        mdadm.mdadm_assemble(md_devname=md_devname, devices=devices)
+        expected_calls = [
+            call(["mdadm", "--assemble", md_devname, "--run"] + devices,
+                 capture=True, rcs=[0, 1, 2]),
+            call(["udevadm", "settle"]),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+
+class TestBlockMdadmCreate(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmCreate, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        # Common mock settings
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'precise'}
+
+    def prepare_mock(self, md_devname, raidlevel, devices, spares):
+        side_effects = []
+        expected_calls = []
+
+        # don't mock anything if raidlevel and spares mismatch
+        if spares and raidlevel not in mdadm.SPARE_RAID_LEVELS:
+            return (side_effects, expected_calls)
+
+        # prepare side-effects
+        for d in devices + spares:
+            side_effects.append(("", ""))  # mdadm --zero-superblock
+            expected_calls.append(
+                call(["mdadm", "--zero-superblock", d], capture=True))
+
+        side_effects.append(("", ""))  # udevadm settle
+        expected_calls.append(call(["udevadm", "settle"]))
+        side_effects.append(("", ""))  # udevadm control --stop-exec-queue
+        expected_calls.append(call(["udevadm", "control",
+                                    "--stop-exec-queue"]))
+        side_effects.append(("", ""))  # mdadm create
+        # build command how mdadm_create does
+        cmd = (["mdadm", "--create", md_devname, "--run",
+               "--level=%s" % raidlevel, "--raid-devices=%s" % len(devices)] +
+               devices)
+        if spares:
+            cmd += ["--spare-devices=%s" % len(spares)] + spares
+
+        expected_calls.append(call(cmd, capture=True))
+        side_effects.append(("", ""))  # udevadm control --start-exec-queue
+        expected_calls.append(call(["udevadm", "control",
+                                    "--start-exec-queue"]))
+        side_effects.append(("", ""))  # udevadm settle
+        expected_calls.append(call(["udevadm", "settle",
+                                    "--exit-if-exists=%s" % md_devname]))
+
+        return (side_effects, expected_calls)
+
+    def test_mdadm_create_raid0(self):
+        md_devname = "/dev/md0"
+        raidlevel = 0
+        devices = ["/dev/vdc1", "/dev/vdd1"]
+        spares = []
+        (side_effects, expected_calls) = self.prepare_mock(md_devname,
+                                                           raidlevel,
+                                                           devices,
+                                                           spares)
+
+        self.mock_util.subp.side_effect = side_effects
+        mdadm.mdadm_create(md_devname=md_devname, raidlevel=raidlevel,
+                           devices=devices, spares=spares)
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+    def test_mdadm_create_raid0_with_spares(self):
+        md_devname = "/dev/md0"
+        raidlevel = 0
+        devices = ["/dev/vdc1", "/dev/vdd1"]
+        spares = ["/dev/vde1"]
+        (side_effects, expected_calls) = self.prepare_mock(md_devname,
+                                                           raidlevel,
+                                                           devices,
+                                                           spares)
+
+        self.mock_util.subp.side_effect = side_effects
+        with self.assertRaises(ValueError):
+            mdadm.mdadm_create(md_devname=md_devname, raidlevel=raidlevel,
+                               devices=devices, spares=spares)
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+    def test_mdadm_create_md_devname_none(self):
+        md_devname = None
+        raidlevel = 0
+        devices = ["/dev/vdc1", "/dev/vdd1"]
+        spares = ["/dev/vde1"]
+        with self.assertRaises(ValueError):
+            mdadm.mdadm_create(md_devname=md_devname, raidlevel=raidlevel,
+                               devices=devices, spares=spares)
+
+    def test_mdadm_create_md_devname_missing(self):
+        self.mock_valid.return_value = False
+        md_devname = "/dev/wark"
+        raidlevel = 0
+        devices = ["/dev/vdc1", "/dev/vdd1"]
+        spares = ["/dev/vde1"]
+        with self.assertRaises(ValueError):
+            mdadm.mdadm_create(md_devname=md_devname, raidlevel=raidlevel,
+                               devices=devices, spares=spares)
+
+    def test_mdadm_create_invalid_raidlevel(self):
+        md_devname = "/dev/md0"
+        raidlevel = 27
+        devices = ["/dev/vdc1", "/dev/vdd1"]
+        spares = ["/dev/vde1"]
+        with self.assertRaises(ValueError):
+            mdadm.mdadm_create(md_devname=md_devname, raidlevel=raidlevel,
+                               devices=devices, spares=spares)
+
+    def test_mdadm_create_check_min_devices(self):
+        md_devname = "/dev/md0"
+        raidlevel = 5
+        devices = ["/dev/vdc1", "/dev/vdd1"]
+        spares = ["/dev/vde1"]
+        with self.assertRaises(ValueError):
+            mdadm.mdadm_create(md_devname=md_devname, raidlevel=raidlevel,
+                               devices=devices, spares=spares)
+
+    def test_mdadm_create_raid5(self):
+        md_devname = "/dev/md0"
+        raidlevel = 5
+        devices = ['/dev/vdc1', '/dev/vdd1', '/dev/vde1']
+        spares = ['/dev/vdg1']
+        (side_effects, expected_calls) = self.prepare_mock(md_devname,
+                                                           raidlevel,
+                                                           devices,
+                                                           spares)
+
+        self.mock_util.subp.side_effect = side_effects
+        mdadm.mdadm_create(md_devname=md_devname, raidlevel=raidlevel,
+                           devices=devices, spares=spares)
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+
+class TestBlockMdadmExamine(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmExamine, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        # Common mock settings
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'precise'}
+
+    def test_mdadm_examine_export(self):
+        self.mock_util.lsb_release.return_value = {'codename': 'xenial'}
+        self.mock_util.subp.return_value = (
+            """
+            MD_LEVEL=raid0
+            MD_DEVICES=2
+            MD_METADATA=0.90
+            MD_UUID=93a73e10:427f280b:b7076c02:204b8f7a
+            """, "")
+
+        device = "/dev/vde"
+        data = mdadm.mdadm_examine(device, export=True)
+
+        expected_calls = [
+            call(["mdadm", "--examine", "--export", device], capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+        self.assertEqual(data['MD_UUID'],
+                         '93a73e10:427f280b:b7076c02:204b8f7a')
+
+    def test_mdadm_examine_no_export(self):
+        self.mock_util.subp.return_value = ("""/dev/vde:
+              Magic : a92b4efc
+            Version : 1.2
+        Feature Map : 0x0
+         Array UUID : 93a73e10:427f280b:b7076c02:204b8f7a
+               Name : wily-foobar:0  (local to host wily-foobar)
+      Creation Time : Sat Dec 12 16:06:05 2015
+         Raid Level : raid1
+       Raid Devices : 2
+
+     Avail Dev Size : 20955136 (9.99 GiB 10.73 GB)
+      Used Dev Size : 20955136 (9.99 GiB 10.73 GB)
+         Array Size : 10477568 (9.99 GiB 10.73 GB)
+        Data Offset : 16384 sectors
+       Super Offset : 8 sectors
+       Unused Space : before=16296 sectors, after=0 sectors
+              State : clean
+        Device UUID : 8fcd62e6:991acc6e:6cb71ee3:7c956919
+
+        Update Time : Sat Dec 12 16:09:09 2015
+      Bad Block Log : 512 entries available at offset 72 sectors
+           Checksum : 65b57c2e - correct
+             Events : 17
+
+
+       Device Role : spare
+       Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
+        """, "")   # mdadm --examine /dev/vde
+
+        device = "/dev/vde"
+        data = mdadm.mdadm_examine(device, export=False)
+
+        expected_calls = [
+            call(["mdadm", "--examine", device], capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+        self.assertEqual(data['MD_UUID'],
+                         '93a73e10:427f280b:b7076c02:204b8f7a')
+
+    def test_mdadm_examine_no_raid(self):
+        self.mock_util.subp.side_effect = subprocess.CalledProcessError("", "")
+
+        device = "/dev/sda"
+        data = mdadm.mdadm_examine(device, export=False)
+
+        expected_calls = [
+            call(["mdadm", "--examine", device], capture=True),
+        ]
+
+        # don't mock anything if raidlevel and spares mismatch
+        self.mock_util.subp.assert_has_calls(expected_calls)
+        self.assertEqual(data, {})
+
+
+class TestBlockMdadmStop(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmStop, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        # Common mock settings
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'xenial'}
+        self.mock_util.subp.side_effect = [
+            ("", ""),  # mdadm stop device
+        ]
+
+    def test_mdadm_stop_no_devpath(self):
+        with self.assertRaises(ValueError):
+            mdadm.mdadm_stop(None)
+
+    def test_mdadm_stop(self):
+        device = "/dev/vdc"
+        mdadm.mdadm_stop(device)
+        expected_calls = [
+            call(["mdadm", "--stop", device], rcs=[0, 1], capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+
+class TestBlockMdadmRemove(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmRemove, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        # Common mock settings
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'xenial'}
+        self.mock_util.subp.side_effect = [
+            ("", ""),  # mdadm remove device
+        ]
+
+    def test_mdadm_remove_no_devpath(self):
+        with self.assertRaises(ValueError):
+            mdadm.mdadm_remove(None)
+
+    def test_mdadm_remove(self):
+        device = "/dev/vdc"
+        mdadm.mdadm_remove(device)
+        expected_calls = [
+            call(["mdadm", "--remove", device], rcs=[0, 1], capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+
+
+class TestBlockMdadmQueryDetail(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmQueryDetail, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        # Common mock settings
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'precise'}
+
+    def test_mdadm_query_detail_export(self):
+        self.mock_util.lsb_release.return_value = {'codename': 'xenial'}
+        self.mock_util.subp.return_value = (
+            """
+            MD_LEVEL=raid1
+            MD_DEVICES=2
+            MD_METADATA=1.2
+            MD_UUID=93a73e10:427f280b:b7076c02:204b8f7a
+            MD_NAME=wily-foobar:0
+            MD_DEVICE_vdc_ROLE=0
+            MD_DEVICE_vdc_DEV=/dev/vdc
+            MD_DEVICE_vdd_ROLE=1
+            MD_DEVICE_vdd_DEV=/dev/vdd
+            MD_DEVICE_vde_ROLE=spare
+            MD_DEVICE_vde_DEV=/dev/vde
+            """, "")
+
+        device = "/dev/md0"
+        self.mock_valid.return_value = True
+        data = mdadm.mdadm_query_detail(device, export=True)
+
+        expected_calls = [
+            call(["mdadm", "--query", "--detail", "--export", device],
+                 capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+        self.assertEqual(data['MD_UUID'],
+                         '93a73e10:427f280b:b7076c02:204b8f7a')
+
+    def test_mdadm_query_detail_no_export(self):
+        self.mock_util.subp.return_value = ("""/dev/md0:
+        Version : 1.2
+  Creation Time : Sat Dec 12 16:06:05 2015
+     Raid Level : raid1
+     Array Size : 10477568 (9.99 GiB 10.73 GB)
+  Used Dev Size : 10477568 (9.99 GiB 10.73 GB)
+   Raid Devices : 2
+  Total Devices : 3
+    Persistence : Superblock is persistent
+
+    Update Time : Sat Dec 12 16:09:09 2015
+          State : clean
+ Active Devices : 2
+Working Devices : 3
+ Failed Devices : 0
+  Spare Devices : 1
+
+           Name : wily-foobar:0  (local to host wily-foobar)
+           UUID : 93a73e10:427f280b:b7076c02:204b8f7a
+         Events : 17
+
+    Number   Major   Minor   RaidDevice State
+       0     253       32        0      active sync   /dev/vdc
+       1     253       48        1      active sync   /dev/vdd
+
+       2     253       64        -      spare   /dev/vde
+        """, "")   # mdadm --query --detail /dev/md0
+
+        device = "/dev/md0"
+        data = mdadm.mdadm_query_detail(device, export=False)
+        expected_calls = [
+            call(["mdadm", "--query", "--detail", device], capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+        self.assertEqual(data['MD_UUID'],
+                         '93a73e10:427f280b:b7076c02:204b8f7a')
+
+
+class TestBlockMdadmDetailScan(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmDetailScan, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        # Common mock settings
+        self.scan_output = ("ARRAY /dev/md0 metadata=1.2 spares=2 name=0 " +
+                            "UUID=b1eae2ff:69b6b02e:1d63bb53:ddfa6e4a")
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'xenial'}
+        self.mock_util.subp.side_effect = [
+            (self.scan_output, ""),  # mdadm --detail --scan
+        ]
+
+    def test_mdadm_remove(self):
+        data = mdadm.mdadm_detail_scan()
+        expected_calls = [
+            call(["mdadm", "--detail", "--scan"], capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+        self.assertEqual(self.scan_output, data)
+
+    def test_mdadm_remove_error(self):
+        self.mock_util.subp.side_effect = [
+            ("wark", "error"),  # mdadm --detail --scan
+        ]
+        data = mdadm.mdadm_detail_scan()
+        expected_calls = [
+            call(["mdadm", "--detail", "--scan"], capture=True),
+        ]
+        self.mock_util.subp.assert_has_calls(expected_calls)
+        self.assertEqual(None, data)
+
+
+class TestBlockMdadmMdHelpers(MdadmTestBase):
+    def setUp(self):
+        super(TestBlockMdadmMdHelpers, self).setUp()
+        self.add_patch('curtin.block.mdadm.util', 'mock_util')
+        self.add_patch('curtin.block.mdadm.is_valid_device', 'mock_valid')
+
+        self.mock_valid.return_value = True
+        self.mock_util.lsb_release.return_value = {'codename': 'xenial'}
+
+    def test_valid_mdname(self):
+        mdname = "/dev/md0"
+        result = mdadm.valid_mdname(mdname)
+        expected_calls = [
+            call(mdname)
+        ]
+        self.mock_valid.assert_has_calls(expected_calls)
+        self.assertTrue(result)
+
+    def test_valid_mdname_short(self):
+        mdname = "md0"
+        result = mdadm.valid_mdname(mdname)
+        expected_calls = [
+            call("/dev/md0")
+        ]
+        self.mock_valid.assert_has_calls(expected_calls)
+        self.assertTrue(result)
+
+    def test_valid_mdname_none(self):
+        mdname = None
+        with self.assertRaises(ValueError):
+            mdadm.valid_mdname(mdname)
+
+    def test_valid_mdname_not_valid_device(self):
+        self.mock_valid.return_value = False
+        mdname = "/dev/md0"
+        with self.assertRaises(ValueError):
+            mdadm.valid_mdname(mdname)
+
+    @patch.object(builtins, "open")
+    def test_md_sysfs_attr(self, mock_open):
+        mdname = "/dev/md0"
+        attr_name = 'array_state'
+        sysfs_path = '/sys/class/block/{}/md/{}'.format(dev_short(mdname),
+                                                        attr_name)
+        mdadm.md_sysfs_attr(mdname, attr_name)
+        mock_open.assert_called_with(sysfs_path)
+
+    def test_md_sysfs_attr_devname_none(self):
+        mdname = None
+        attr_name = 'array_state'
+        with self.assertRaises(ValueError):
+            mdadm.md_sysfs_attr(mdname, attr_name)
+
+    def test_md_raidlevel_short(self):
+        for rl in [0, 1, 5, 6, 10, 'linear', 'stripe']:
+            self.assertEqual(rl, mdadm.md_raidlevel_short(rl))
+            if isinstance(rl, int):
+                long_rl = 'raid%d' % rl
+                self.assertEqual(rl, mdadm.md_raidlevel_short(long_rl))
+
+    def test_md_minimum_devices(self):
+        min_to_rl = {
+            2: [0, 1, 'linear', 'stripe'],
+            3: [5],
+            4: [6, 10],
+        }
+
+        for rl in [0, 1, 5, 6, 10, 'linear', 'stripe']:
+            min_devs = mdadm.md_minimum_devices(rl)
+            self.assertTrue(rl in min_to_rl[min_devs])
+
+    def test_md_minimum_devices_invalid_rl(self):
+        min_devs = mdadm.md_minimum_devices(27)
+        self.assertEqual(min_devs, -1)
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_rw(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.return_value = 'clean'
+        self.assertTrue(mdadm.md_check_array_state_rw(mdname))
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_rw_false(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.return_value = 'inactive'
+        self.assertFalse(mdadm.md_check_array_state_rw(mdname))
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_ro(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.return_value = 'readonly'
+        self.assertTrue(mdadm.md_check_array_state_ro(mdname))
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_ro_false(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.return_value = 'inactive'
+        self.assertFalse(mdadm.md_check_array_state_ro(mdname))
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_error(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.return_value = 'inactive'
+        self.assertTrue(mdadm.md_check_array_state_error(mdname))
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_error_false(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.return_value = 'active'
+        self.assertFalse(mdadm.md_check_array_state_error(mdname))
+
+    def test_md_device_key_role(self):
+        devname = '/dev/vda'
+        rolekey = mdadm.md_device_key_role(devname)
+        self.assertEqual('MD_DEVICE_vda_ROLE', rolekey)
+
+    def test_md_device_key_role_no_dev(self):
+        devname = None
+        with self.assertRaises(ValueError):
+            mdadm.md_device_key_role(devname)
+
+    def test_md_device_key_dev(self):
+        devname = '/dev/vda'
+        devkey = mdadm.md_device_key_dev(devname)
+        self.assertEqual('MD_DEVICE_vda_DEV', devkey)
+
+    def test_md_device_key_dev_no_dev(self):
+        devname = None
+        with self.assertRaises(ValueError):
+            mdadm.md_device_key_dev(devname)
+
+    @patch('curtin.block.mdadm.os.path.exists')
+    @patch('curtin.block.mdadm.os.listdir')
+    def tests_md_get_spares_list(self, mock_listdir, mock_exists):
+        mdname = '/dev/md0'
+        devices = ['dev-vda', 'dev-vdb', 'dev-vdc']
+        states = ['in-sync', 'in-sync', 'spare']
+
+        mock_exists.return_value = True
+        mock_listdir.return_value = devices
+        self.mock_util.load_file.side_effect = states
+
+        sysfs_path = '/sys/class/block/md0/md/'
+
+        expected_calls = []
+        for d in devices:
+            expected_calls.append(call(os.path.join(sysfs_path, d, 'state')))
+
+        spares = mdadm.md_get_spares_list(mdname)
+        self.mock_util.load_file.assert_has_calls(expected_calls)
+        self.assertEqual(['/dev/vdc'], spares)
+
+    @patch('curtin.block.mdadm.os.path.exists')
+    def tests_md_get_spares_list_nomd(self, mock_exists):
+        mdname = '/dev/md0'
+        mock_exists.return_value = False
+        with self.assertRaises(ValueError):
+            mdadm.md_get_spares_list(mdname)
+
+    @patch('curtin.block.mdadm.os.path.exists')
+    @patch('curtin.block.mdadm.os.listdir')
+    def tests_md_get_devices_list(self, mock_listdir, mock_exists):
+        mdname = '/dev/md0'
+        devices = ['dev-vda', 'dev-vdb', 'dev-vdc']
+        states = ['in-sync', 'in-sync', 'spare']
+
+        mock_exists.return_value = True
+        mock_listdir.return_value = devices
+        self.mock_util.load_file.side_effect = states
+
+        sysfs_path = '/sys/class/block/md0/md/'
+
+        expected_calls = []
+        for d in devices:
+            expected_calls.append(call(os.path.join(sysfs_path, d, 'state')))
+
+        devs = mdadm.md_get_devices_list(mdname)
+        self.mock_util.load_file.assert_has_calls(expected_calls)
+        self.assertEqual(sorted(['/dev/vda', '/dev/vdb']), sorted(devs))
+
+    @patch('curtin.block.mdadm.os.path.exists')
+    def tests_md_get_devices_list_nomd(self, mock_exists):
+        mdname = '/dev/md0'
+        mock_exists.return_value = False
+        with self.assertRaises(ValueError):
+            mdadm.md_get_devices_list(mdname)
+
+    @patch('curtin.block.mdadm.os')
+    def test_md_check_array_uuid(self, mock_os):
+        devname = '/dev/md0'
+        md_uuid = '93a73e10:427f280b:b7076c02:204b8f7a'
+        mock_os.path.realpath.return_value = devname
+        rv = mdadm.md_check_array_uuid(devname, md_uuid)
+        self.assertTrue(rv)
+
+    @patch('curtin.block.mdadm.os')
+    def test_md_check_array_uuid_mismatch(self, mock_os):
+        devname = '/dev/md0'
+        md_uuid = '93a73e10:427f280b:b7076c02:204b8f7a'
+        mock_os.path.realpath.return_value = '/dev/md1'
+
+        with self.assertRaises(ValueError):
+            mdadm.md_check_array_uuid(devname, md_uuid)
+
+    @patch('curtin.block.mdadm.mdadm_query_detail')
+    def test_md_get_uuid(self, mock_query):
+        mdname = '/dev/md0'
+        md_uuid = '93a73e10:427f280b:b7076c02:204b8f7a'
+        mock_query.return_value = {'MD_UUID': md_uuid}
+        uuid = mdadm.md_get_uuid(mdname)
+        self.assertEqual(md_uuid, uuid)
+
+    @patch('curtin.block.mdadm.mdadm_query_detail')
+    def test_md_get_uuid_dev_none(self, mock_query):
+        mdname = None
+        with self.assertRaises(ValueError):
+            mdadm.md_get_uuid(mdname)
+
+    def test_md_check_raid_level(self):
+        for rl in mdadm.VALID_RAID_LEVELS:
+            self.assertTrue(mdadm.md_check_raidlevel(rl))
+
+    def test_md_check_raid_level_bad(self):
+        bogus = '27'
+        self.assertTrue(bogus not in mdadm.VALID_RAID_LEVELS)
+        with self.assertRaises(ValueError):
+            mdadm.md_check_raidlevel(bogus)
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.side_effect = [
+            'clean',  # array_state
+            '0',  # degraded
+            'idle',  # sync_action
+        ]
+        self.assertTrue(mdadm.md_check_array_state(mdname))
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_norw(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.side_effect = [
+            'suspended',  # array_state
+            '0',  # degraded
+            'idle',  # sync_action
+        ]
+        with self.assertRaises(ValueError):
+            mdadm.md_check_array_state(mdname)
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_degraded(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.side_effect = [
+            'clean',  # array_state
+            '1',  # degraded
+            'idle',  # sync_action
+        ]
+        with self.assertRaises(ValueError):
+            mdadm.md_check_array_state(mdname)
+
+    @patch('curtin.block.mdadm.md_sysfs_attr')
+    def test_md_check_array_state_sync(self, mock_attr):
+        mdname = '/dev/md0'
+        mock_attr.side_effect = [
+            'clean',  # array_state
+            '0',  # degraded
+            'recovery',  # sync_action
+        ]
+        with self.assertRaises(ValueError):
+            mdadm.md_check_array_state(mdname)
+
+    @patch('curtin.block.mdadm.md_check_array_uuid')
+    @patch('curtin.block.mdadm.md_get_uuid')
+    def test_md_check_uuid(self, mock_guuid, mock_ckuuid):
+        mdname = '/dev/md0'
+        mock_guuid.return_value = '93a73e10:427f280b:b7076c02:204b8f7a'
+        mock_ckuuid.return_value = True
+
+        rv = mdadm.md_check_uuid(mdname)
+        self.assertTrue(rv)
+
+    @patch('curtin.block.mdadm.md_check_array_uuid')
+    @patch('curtin.block.mdadm.md_get_uuid')
+    def test_md_check_uuid_nouuid(self, mock_guuid, mock_ckuuid):
+        mdname = '/dev/md0'
+        mock_guuid.return_value = None
+        with self.assertRaises(ValueError):
+            mdadm.md_check_uuid(mdname)
+
+    @patch('curtin.block.mdadm.md_get_devices_list')
+    def test_md_check_devices(self, mock_devlist):
+        mdname = '/dev/md0'
+        devices = ['/dev/vdc', '/dev/vdd']
+
+        mock_devlist.return_value = devices
+        rv = mdadm.md_check_devices(mdname, devices)
+        self.assertEqual(rv, None)
+
+    @patch('curtin.block.mdadm.md_get_devices_list')
+    def test_md_check_devices_wrong_devs(self, mock_devlist):
+        mdname = '/dev/md0'
+        devices = ['/dev/vdc', '/dev/vdd']
+
+        mock_devlist.return_value = ['/dev/sda']
+        with self.assertRaises(ValueError):
+            mdadm.md_check_devices(mdname, devices)
+
+    def test_md_check_devices_no_devs(self):
+        mdname = '/dev/md0'
+        devices = []
+
+        with self.assertRaises(ValueError):
+            mdadm.md_check_devices(mdname, devices)
+
+    @patch('curtin.block.mdadm.md_get_spares_list')
+    def test_md_check_spares(self, mock_devlist):
+        mdname = '/dev/md0'
+        spares = ['/dev/vdc', '/dev/vdd']
+
+        mock_devlist.return_value = spares
+        rv = mdadm.md_check_spares(mdname, spares)
+        self.assertEqual(rv, None)
+
+    @patch('curtin.block.mdadm.md_get_spares_list')
+    def test_md_check_spares_wrong_devs(self, mock_devlist):
+        mdname = '/dev/md0'
+        spares = ['/dev/vdc', '/dev/vdd']
+
+        mock_devlist.return_value = ['/dev/sda']
+        with self.assertRaises(ValueError):
+            mdadm.md_check_spares(mdname, spares)
+
+    @patch('curtin.block.mdadm.mdadm_examine')
+    @patch('curtin.block.mdadm.mdadm_query_detail')
+    @patch('curtin.block.mdadm.md_get_uuid')
+    def test_md_check_array_membership(self, mock_uuid, mock_query,
+                                       mock_examine):
+        mdname = '/dev/md0'
+        devices = ['/dev/vda', '/dev/vdb', '/dev/vdc', '/dev/vdd']
+        md_uuid = '93a73e10:427f280b:b7076c02:204b8f7a'
+        md_dict = {'MD_UUID': md_uuid}
+        mock_query.return_value = md_dict
+        mock_uuid.return_value = md_uuid
+        mock_examine.side_effect = [md_dict] * len(devices)
+        expected_calls = []
+        for dev in devices:
+            expected_calls.append(call(dev, export=False))
+
+        rv = mdadm.md_check_array_membership(mdname, devices)
+
+        self.assertEqual(rv, None)
+        mock_uuid.assert_has_calls([call(mdname)])
+        mock_examine.assert_has_calls(expected_calls)
+
+    @patch('curtin.block.mdadm.mdadm_examine')
+    @patch('curtin.block.mdadm.mdadm_query_detail')
+    @patch('curtin.block.mdadm.md_get_uuid')
+    def test_md_check_array_membership_bad_dev(self, mock_uuid, mock_query,
+                                               mock_examine):
+        mdname = '/dev/md0'
+        devices = ['/dev/vda', '/dev/vdb', '/dev/vdc', '/dev/vdd']
+        md_uuid = '93a73e10:427f280b:b7076c02:204b8f7a'
+        md_dict = {'MD_UUID': md_uuid}
+        mock_query.return_value = md_dict
+        mock_uuid.return_value = md_uuid
+        mock_examine.side_effect = [
+            md_dict,
+            {},
+            md_dict,
+            md_dict,
+        ]  # one device isn't a member
+
+        with self.assertRaises(ValueError):
+            mdadm.md_check_array_membership(mdname, devices)
+
+    @patch('curtin.block.mdadm.mdadm_examine')
+    @patch('curtin.block.mdadm.mdadm_query_detail')
+    @patch('curtin.block.mdadm.md_get_uuid')
+    def test_md_check_array_membership_wrong_array(self, mock_uuid, mock_query,
+                                                   mock_examine):
+        mdname = '/dev/md0'
+        devices = ['/dev/vda', '/dev/vdb', '/dev/vdc', '/dev/vdd']
+        md_uuid = '93a73e10:427f280b:b7076c02:204b8f7a'
+        md_dict = {'MD_UUID': '11111111:427f280b:b7076c02:204b8f7a'}
+        mock_query.return_value = md_dict
+        mock_uuid.return_value = md_uuid
+        mock_examine.side_effect = [md_dict] * len(devices)
+
+        with self.assertRaises(ValueError):
+            mdadm.md_check_array_membership(mdname, devices)
+
+    @patch('curtin.block.mdadm.md_check_array_membership')
+    @patch('curtin.block.mdadm.md_check_spares')
+    @patch('curtin.block.mdadm.md_check_devices')
+    @patch('curtin.block.mdadm.md_check_uuid')
+    @patch('curtin.block.mdadm.md_check_raidlevel')
+    @patch('curtin.block.mdadm.md_check_array_state')
+    def test_md_check_all_good(self, mock_array, mock_raid, mock_uuid,
+                               mock_dev, mock_spare, mock_member):
+        md_devname = '/dev/md0'
+        raidlevel = 1
+        devices = ['/dev/vda', '/dev/vdb']
+        spares = ['/dev/vdc']
+
+        mock_array.return_value = None
+        mock_raid.return_value = None
+        mock_uuid.return_value = None
+        mock_dev.return_value = None
+        mock_spare.return_value = None
+        mock_member.return_value = None
+
+        mdadm.md_check(md_devname, raidlevel, devices=devices, spares=spares)
+
+        mock_array.assert_has_calls([call(md_devname)])
+        mock_raid.assert_has_calls([call(raidlevel)])
+        mock_uuid.assert_has_calls([call(md_devname)])
+        mock_dev.assert_has_calls([call(md_devname, devices)])
+        mock_spare.assert_has_calls([call(md_devname, spares)])
+        mock_member.assert_has_calls([call(md_devname, devices + spares)])
+
+# vi: ts=4 expandtab syntax=python