Merge lp:~wesley-wiedenmeier/curtin/partial-testing into lp:~curtin-dev/curtin/trunk

[Wesley Wiedenmeier]

Add storagetest test suite to isolate storage configuration

[Wesley Wiedenmeier (wesley-wiedenmeier)]

The storagetests are now at a point where they can be fully run and the results collected just using 'make run_storagetests'. The storagetest_runner uses the same general structure as vmtests so it shouldn't require any modifications of the test system other than adding it to the list of things to run.

The storagetest_runner generates a tarball of curtin and issues cloud-config scripts to the target image to retrieve and extract that tarball as it boots. The storagetests are then run, and use curtin.reporter to report success and failure to the test runner, as well as log files generated by block_meta while the test was running

[Wesley Wiedenmeier (wesley-wiedenmeier)]

I think that the storagetests and the storagetest_runner are ready for review now, this added a lot of verification for storage configuration and makes it easy to add quite a bit more. The test runner works pretty well, and I have managed to run all of the tests in about 2 hours with 4 proceses. I think that quite a bit of the time there was used waiting for apt-get on a bad internet connection. I haven't tried with a local repo mirror yet though.

There are still some outstanding issues running some of the actual test cases, but these will require bugfixes on curtin to correct. The test cases which cannot be run right now are disabled in the test suite, and can be enabled when the bugs they cause are fixed.

There are some other branches pending merge that this branch depends on, I will go through and make a list of those tomorrow.

I also added a doc to doc/devel/ about storagetests and storagetest_runner that was based on the doc for vmtests.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:470
https://server-team-jenkins.canonical.com/job/curtin-ci/303/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/300/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/303/rebuild

review: Approve (continuous-integration)

[Christian Ehrhardt  (paelzer)]

Hi Wesley,

I just ran your branch to take a look and found some minor things I'd ask you if you consider them useful to add.

The json files - while being json and not intended for humans - would still be much more readable it they had some line breaks. Like what "python -m json.tool /tmp/foo/storage-events.json" gives me would be great.

Then in the log a few outputs seem not to go through LOG.
I have a few questions about some of them:

1. Parse storage tests reporting data and ensure that all tests passed ... ok
Should get a "2016-07-12 07:15:09,184 - vmtests - INFO -" prefix like the others IMHO.

2. 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-reporting.json HTTP/1.1" 200 -
10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-disk-conf.json HTTP/1.1" 200 -
10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz HTTP/1.1" 200 -

What are thos actually doing, it is always the same text - Could be something like:
2016-07-12 07:15:09,184 - vmtests - INFO - fetching json reporting
2016-07-12 07:15:09,184 - vmtests - INFO - fetching json disk config
2016-07-12 07:15:09,184 - vmtests - INFO - fetching foo tarball

3. just after "10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz HTTP/1.1" 200 -" is the longest "wait" duration in the tests. There should be some sort of "doing the test now" or anything else that somebody looking at it knows what is taking the time. This doesn't have to be step 1,2,3,4,../20 - just after the "Booting target image" a short "running test XY now".

[Christian Ehrhardt  (paelzer)]

Some comments when reading through the code

[Christian Ehrhardt  (paelzer)]

For a speed check I tried to see if it has issues going concurrently.
I think I didn't find the right way to call it - I tried:
rm -rf output/; CURTIN_VMTEST_KEEP_DATA_FAIL=all ./tools/jenkins-runner -vv --nologcapture --processes=10 --process-timeout=3600 tests/storagetest_runner/

I found that with this call the section following section tries to write an infinitely huge file:
Building tarball of curtin: /mnt/nvme/curtin-wesley/partial-testing

Wanted to let you know just in case that infinite write is a bug.

Later on I found this in the doc:
nosetests3 --processes=-1 tests/storagetest_runner

That gave me a stuck system - maybe too much cpus (6x2threads) and by that too much output?
In any case the output indenting was totally broken - I had to reset my console to scroll again.

That failed me then with errors, not sure if that is bad or just a wrong call - here is the log: http://paste.ubuntu.com/19157752/
I realized that since this first "hanging" processes -1 run all tests were failing this way now.
All on:
Traceback (most recent call last):
  File "/mnt/nvme/curtin-wesley/partial-testing/tests/storagetest_runner/__init__.py", line 385, in test_reported_results
    self.assertTrue(os.path.isfile(self.storage_log))
AssertionError: False is not true

Debugging gave me: "(qemu) qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory"
That likely also was my first hanging - but that could be fixed by freeing some up :-)
But I wonder if we need some sort of "is enough mem avail" prior to call qemu?

In the following retry it left me again with a good return code, but plenty of running qemu processes up.
That really needs some hardening.

Then I wanted to step down and did only:
nosetests3 --processes=2 tests/storagetest_runner
To check if it works at all.
I got the same console that gets misformatted after a while ending with
Ran 0 tests in 112.494s

Since all fails keep the logs around here the log file: http://paste.ubuntu.com/19158435/

TL;DR: concurrent execution needs some fixes and probably a bit hardening against shooting itself :-)

[Christian Ehrhardt  (paelzer)]

I ran checkers and found several of the following (no need to mark them all inline):

tests/storagetest_runner/__init__.py|101 col 16 warning| [unused-variable] Unused variable '_err' [python/pylint]

=> if really unused a _ would be even better.

tests/storagetest_runner/__init__.py|208 col 18 warning| [logging-format-interpolation] Use % formatting in logging functions and pass the % parameters as arguments

=> since log would format for you ...

I know pylint it is noisy and sometimes even disagrees with other checkers, but most of these are also only a search and replace away so probably worth to fix.

I'm not so sure about
tests/storagetests/__init__.py|186 col 16 warning| [unidiomatic-typecheck] Using type() instead of isinstance() for a typecheck. [python/pylint]

tests/storagetests/__init__.py|219 col 24 warning| [redefined-builtin] Redefining built-in 'type' [python/pylint]

I already complained about short names before, this is another example (there are more like mp, fp, e, ...)
tests/storagetests/__init__.py|224 col 13 warning| [invalid-name] Invalid variable name "d" [python/pylint]

A totally different one is:
tests/storagetests/verifiers.py|15 col 13 error| [no-member] Instance of 'BasicTests' has no 'assertIsNotNone' member [python/pylint]
That never is an issue so far, as all inheriting classes of e.g. BasicTests also inherit from class BaseStorageTest(unittest.TestCase). But to make sure the methods are in scope would it hurt letting the verifiers also derive from unittest.TestCase?

One that is probably worth for sure to avoid later issues:
tests/storagetests/__init__.py|82 col 31 warning| [redefined-outer-name] Redefining name 'reporter' from outer scope (line 2) [python/pylint]

[Wesley Wiedenmeier (wesley-wiedenmeier)]

Hey, thanks for looking through and reviewing, I'm reading through the diff comments atm, I'll reply inline.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

> For a speed check I tried to see if it has issues going concurrently.
> I think I didn't find the right way to call it - I tried:
> rm -rf output/; CURTIN_VMTEST_KEEP_DATA_FAIL=all ./tools/jenkins-runner -vv
> --nologcapture --processes=10 --process-timeout=3600 tests/storagetest_runner/
>
> I found that with this call the section following section tries to write an
> infinitely huge file:
> Building tarball of curtin: /mnt/nvme/curtin-wesley/partial-testing
>
> Wanted to let you know just in case that infinite write is a bug.

Thanks for pointing that out, that's definintely a bug in how I generate the curtin tarball. I hadn't run before in the jenkins runner, because sparse files don't work right in my /home partition and I didn't want the tests writing huge files there, but yeah, the jenkins runner puts the curtin tmp dir inside the curtin dir being tested, so it would cause tar to try to include the tarball its generating in itself. I need to make the tests aware of what environment they're running in so they know to omit that directory.

> Later on I found this in the doc:
> nosetests3 --processes=-1 tests/storagetest_runner
>
> That gave me a stuck system - maybe too much cpus (6x2threads) and by that too
> much output?
> In any case the output indenting was totally broken - I had to reset my
> console to scroll again.

Yeah, I've seen the output get messed up too, but I think what is happening is just that the processes are racing to write to stdout and the data they're writing is getting corrupted somehow. I haven't tried with more than 4 processes at once, so there may be some bugs that occur in that case. I'll look through the log and try to see if I can figure out what was going wrong there.

> That failed me then with errors, not sure if that is bad or just a wrong call
> - here is the log: http://paste.ubuntu.com/19157752/
> I realized that since this first "hanging" processes -1 run all tests were
> failing this way now.
> All on:
> Traceback (most recent call last):
> File "/mnt/nvme/curtin-wesley/partial-
> testing/tests/storagetest_runner/__init__.py", line 385, in
> test_reported_results
> self.assertTrue(os.path.isfile(self.storage_log))
> AssertionError: False is not true
>
> Debugging gave me: "(qemu) qemu-system-x86_64: cannot set up guest memory
> 'pc.ram': Cannot allocate memory"
> That likely also was my first hanging - but that could be fixed by freeing
> some up :-)
> But I wonder if we need some sort of "is enough mem avail" prior to call qemu?

Yeah, the test runner eats up memory, not quite as much as the vmtests do while the target system tarball is being extracted in the vm though. A check would definitely be good, I'll look into how to write that.

Watching a vm trying to run from swap isn't fun :)

> In the following retry it left me again with a good return code, but plenty of
> running qemu processes up.
> That really needs some hardening.
>
> Then I wanted to step down and did only:
> nosetests3 --processes=2 tests/storagetest_runner
> To check if it works at all.
> I got the same console that gets misformatted after a while ending with
> Ran 0 tests in 112.494s
>
> Since all fails keep th...

[Wesley Wiedenmeier (wesley-wiedenmeier)]

> Hi Wesley,
>
> I just ran your branch to take a look and found some minor things I'd ask you
> if you consider them useful to add.
>
> The json files - while being json and not intended for humans - would still be
> much more readable it they had some line breaks. Like what "python -m
> json.tool /tmp/foo/storage-events.json" gives me would be great.

Yeah, that makes sense. I'll switch it to use something like 'json.dump(... indent=4)'

> Then in the log a few outputs seem not to go through LOG.
> I have a few questions about some of them:
>
> 1. Parse storage tests reporting data and ensure that all tests passed ... ok
> Should get a "2016-07-12 07:15:09,184 - vmtests - INFO -" prefix like the
> others IMHO.

Oh, that's the docstring for the test function, I think that's just added to nosetests's output when running with '-vv'. It may be good to emit a message through LOG though and remove the docstring cause the times could be good for debugging.

> 2. 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-reporting.json
> HTTP/1.1" 200 -
> 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /storagetest-disk-conf.json
> HTTP/1.1" 200 -
> 10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz HTTP/1.1" 200 -
>
> What are thos actually doing, it is always the same text - Could be something
> like:
> 2016-07-12 07:15:09,184 - vmtests - INFO - fetching json reporting
> 2016-07-12 07:15:09,184 - vmtests - INFO - fetching json disk config
> 2016-07-12 07:15:09,184 - vmtests - INFO - fetching foo tarball

I'm not sure if I can actually change that or not, those messages come from http.server, and I don't know if there is an interface in it to handle custom messages, I'll check in the source for it though.

> 3. just after "10.245.168.14 - - [12/Jul/2016 07:15:22] "GET /curtin.tar.xz
> HTTP/1.1" 200 -" is the longest "wait" duration in the tests. There should be
> some sort of "doing the test now" or anything else that somebody looking at it
> knows what is taking the time. This doesn't have to be step 1,2,3,4,../20 -
> just after the "Booting target image" a short "running test XY now".

Yeah, it would definitely be nice to have a bit of debug there. I can add in optional logging to tools.report_webhook_logger to handle outputting events to stdio as well as the logfile when they're received, that should work okay

[Wesley Wiedenmeier (wesley-wiedenmeier)]

> I ran checkers and found several of the following (no need to mark them all
> inline):
>
> tests/storagetest_runner/__init__.py|101 col 16 warning| [unused-variable]
> Unused variable '_err' [python/pylint]
>
> => if really unused a _ would be even better.

Yeah, that makes sense. I just got in the habit of writing _err from block meta, but I should probably clean up some of the variable names there too. For the most part the err part of the util.subp return is never used.

> tests/storagetest_runner/__init__.py|208 col 18 warning| [logging-format-
> interpolation] Use % formatting in logging functions and pass the % parameters
> as arguments
>
> => since log would format for you ...
>
> I know pylint it is noisy and sometimes even disagrees with other checkers,
> but most of these are also only a search and replace away so probably worth to
> fix.

Yeah, I can use the old style format specifiers there, I'll switch that over.

> I'm not so sure about
> tests/storagetests/__init__.py|186 col 16 warning| [unidiomatic-typecheck]
> Using type() instead of isinstance() for a typecheck. [python/pylint]

Right yeah, I'll switch that, isinstance reads nicer.

> tests/storagetests/__init__.py|219 col 24 warning| [redefined-builtin]
> Redefining built-in 'type' [python/pylint]

Oh, yeah, that was for error type in __exit__, I'll rename that to 'etype'

> I already complained about short names before, this is another example (there
> are more like mp, fp, e, ...)
> tests/storagetests/__init__.py|224 col 13 warning| [invalid-name] Invalid
> variable name "d" [python/pylint]

Yeah, I think some of these are just trying to keep the line from going over 80, but some were probably in the list comprehensions, where the scope of the var is so limited it should be okay. I'll switch everything outside of comprehensions over to long names real quick, thanks for bringing that up.

>
> A totally different one is:
> tests/storagetests/verifiers.py|15 col 13 error| [no-member] Instance of
> 'BasicTests' has no 'assertIsNotNone' member [python/pylint]
> That never is an issue so far, as all inheriting classes of e.g. BasicTests
> also inherit from class BaseStorageTest(unittest.TestCase). But to make sure
> the methods are in scope would it hurt letting the verifiers also derive from
> unittest.TestCase?

I guess the verifiers could inherit from TestCase, but I think it may be cleaner for them not to, partly just because that would require '__test__ = False' in all of them to prevent nosetests3 from trying to run them as individual tests. Also, because the BaseStorageTest class overrides the TestCase.run() method, if the verifier classes were to inherit from TestCase and were listed as parents of the individual test classes after the BaseStorageTest, I think that the call to super() in BaseStorageTest.run() would mess up, but I'm not 100% sure there. I should probably do something like define an __init__ for the verifier classes that checks something like 'if not isinstance(self, TestCase)' and throws an error to make the verifiers unusable outside of the test classes.

> One that is probably worth for sure to avoid later issues:
> tests/storagetests/__init__.p...

[Wesley Wiedenmeier (wesley-wiedenmeier)]

Replied to some of the inline commments

[lp:~wesley-wiedenmeier/curtin/partial-testing]

471. By Wesley Wiedenmeier on 2016-07-12

Fix typo in README-storagetests.txt

472. By Wesley Wiedenmeier on 2016-07-12

Merge in more recent revision of lp:~wesley-wiedenmeier/curtin/trusty-preserve
for cleaner handling of blkid failure in disk_handler preserve code on trusty

473. By Wesley Wiedenmeier on 2016-07-12

Use util.json_dumps instead of json.dumps for json formatting for storage tests
as it sets indentation levels nicely

474. By Wesley Wiedenmeier on 2016-07-12

replaced docstring in storagetest_runner.test_reporting_data with LOG.info
message to get a timestap on the output from it in nosetests

475. By Wesley Wiedenmeier on 2016-07-13

Cleanups based on pylint

476. By Wesley Wiedenmeier on 2016-07-13

In storagetest_runner, when building curtin tarball, exclude the 'output' dir
so that tar does not try to create a tarball recursively when using jenkins
runner

[Wesley Wiedenmeier (wesley-wiedenmeier)]

I just pushed some cleanup for the comments posted. Not everything's been addressed yet, but the rest I'll get to tomorrow.

Just so it doesn't get lost, the diff comments are on revision 470.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:476
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/8/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/303/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/8/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

477. By Wesley Wiedenmeier on 2016-07-13

Supress server.SimpleHTTPRequestHandler logging messages

478. By Wesley Wiedenmeier on 2016-07-13

some cleanup in block based on diff comments

479. By Wesley Wiedenmeier on 2016-07-13

Re merge from lp:~wesley-wiedenmeier/curtin/curtin-fix-sysfs-partition-data to
pull in cleaner variable handling

480. By Wesley Wiedenmeier on 2016-07-13

better handling of test_py_ver in storagetest_runner.gen_user_data

481. By Wesley Wiedenmeier on 2016-07-13

sp

482. By Wesley Wiedenmeier on 2016-07-13

Updated storagetest_runner documentation with complete list of environment
variables used in vmtests that are still applicable to storagetest_runner

483. By Wesley Wiedenmeier on 2016-07-14

Wait until enough memory is available before starting tests

 - tools/launch: in Usage() state that --mem arg is in Mb not Kb, as this is
   what qemu takes as input
 - vmtests/__init__.py:
   - add the function stall_if_not_enough_memory()
     this function gets free memory from /proc/meminfo and if less than what is
     required, delays then tries again up to a configurable maximum amount of
     time until enough memory is available to start the tests. if the maximum
     amount of delay time is reached and there still is not enough memory
     available it raises an error that stops the vmtest
   - add environment configuration variables CURTIN_VMTEST_INSTANCE_MEMORY and
     CURTIN_VMTEST_MEMORY_MAX_STALL to control how much memory to allocate to
     the test vm and how long to wait until there is enough memory available
   - for both the tools/launch and tools/xkvm command, specify how much memory
     qemu should use based on the value of CURTIN_VMTEST_INSTANCE_MEMORY
 - storagetest_runner/__init__.py: use the stall_if_not_enough_memory()
   function from vmtests and specify how much memory used based on the vmtest
   environment variable

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:483
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/10/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/306/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/10/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:483
https://server-team-jenkins.canonical.com/job/curtin-ci/305/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/308/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/305/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

484. By Wesley Wiedenmeier on 2016-07-18

Add trusty test interactive

485. By Wesley Wiedenmeier on 2016-07-18

Enable dos logical/extended pattitioning test

486. By Wesley Wiedenmeier on 2016-07-18

Enable trusty storagetest_runner instances as lp: #1596384 only occurs under
really heavy loads on a system without enough resources to run the tests
properly, and should not affect the test server

487. By Wesley Wiedenmeier on 2016-07-18

Instead of parsing /proc/meminfo for vmtests.stall_if_not_enough_memory, use
'free -m'

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:487
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/17/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/321/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/17/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:487
https://server-team-jenkins.canonical.com/job/curtin-ci/313/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/322/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/313/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

i really like the goal here.
some comments inline.

[lp:~wesley-wiedenmeier/curtin/partial-testing]

488. By Wesley Wiedenmeier on 2016-07-18

Remove -w flag from free cmd in stall_if_not_enough_memory

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:488
https://server-team-jenkins.canonical.com/job/curtin-ci/314/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/323/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/314/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:488
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/18/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/324/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/18/rebuild

review: Approve (continuous-integration)

[Wesley Wiedenmeier (wesley-wiedenmeier)]

Thanks for reviewing. I went through and replied to the diff comments, I'll get most of them handled soon, one or two may take a bit longer. Just so they aren't lost because of future commits, the diff comments are at: r487

[lp:~wesley-wiedenmeier/curtin/partial-testing]

489. By Wesley Wiedenmeier on 2016-07-18

Merge from lp:~wesley-wiedenmeier/curtin/trusty-preserve to get better
formatting of log messages in disk_handler without using \

490. By Wesley Wiedenmeier on 2016-07-18

Use encode=False rather than .decode() in image sync output

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:490
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/20/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/327/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/20/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:490
https://server-team-jenkins.canonical.com/job/curtin-ci/316/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/328/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/316/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

491. By Wesley Wiedenmeier on 2016-07-20

renamed some of the storagetests configs with non descriptive names

492. By Wesley Wiedenmeier on 2016-07-20

In storagetest_runner, don't use make to start tests, use nosetests directly,
to avoid having to install make

493. By Wesley Wiedenmeier on 2016-07-20

Remove encoding from util.json_dumps as it isn't actually needed anywhere
where util.json_dumps is used

494. By Wesley Wiedenmeier on 2016-07-20

Fixed call to util.json_dumps

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:494
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/24/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/334/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/24/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:494
https://server-team-jenkins.canonical.com/job/curtin-ci/319/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/335/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/319/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

495. By Wesley Wiedenmeier on 2016-07-22

Removed vmtests.stall_if_not_enough memory and associated varialbes and
documentation, as it has been moved into another branch to merge separately

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:495
https://server-team-jenkins.canonical.com/job/curtin-ci/325/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/346/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/325/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:495
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/30/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/347/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/30/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

496. By Wesley Wiedenmeier on 2016-07-22

Merge from lp:~wesley-wiedenmeier/curtin/trunk.add-web-reporter-to-vmtests

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:496
https://server-team-jenkins.canonical.com/job/curtin-ci/326/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/348/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/326/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:496
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/31/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/349/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/31/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

497. By Wesley Wiedenmeier on 2016-07-22

Merge from trunk to resole conflict

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:497
https://server-team-jenkins.canonical.com/job/curtin-ci/327/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/350/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/327/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:497
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/32/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/351/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/32/rebuild

review: Approve (continuous-integration)

[Wesley Wiedenmeier (wesley-wiedenmeier)]

There are some branches that have already been merged into
this branch because they were neccessary for getting
storagetests working at all. Most of them are bug fixes,
except trunk.add-web-reporter-to-vmtests, which introduces
report_webhook_logger.CaptureReporting, to retrieve curtin
reporting from a test environment.

In addition, several bugs have already been found using the
current test configs for storagetests. Branches exist to
handle most of these. Those branches have not been merged
into this one, but once they have merged into trunk I can
enable the storagetests that reproduce those bugs.

Already Merged Branches:
lp:~wesley-wiedenmeier/curtin/trunk.add-web-reporter-to-vmtests
    merge into: lp:~raharper/curtin/trunk.add-web-reporter-to-vmtests
    reason: improvements to original logic for
            CaptureReporting and get_lan_ip, makes it
            possible to use tests with reporting in
            parallel by getting port number to bind to
            dynamically and uses ifconfig instead of
            'ip address' to get local lan ip in vmtests
lp:~raharper/curtin/trunk.add-web-reporter-to-vmtests
    merge into: lp:curtin
    reason: during vmtests, capture reporting events from
            curtin and verify that the events are recieved
            and properly formatted
lp:~wesley-wiedenmeier/curtin/curtin-fix-sysfs-partition-data
    merge info: lp:curtin
    reason: the block.sysfs_partition_data function is
            useful, and is used in storagetests, but the
            old implementation was only able to operate on
            the path to whole disks, not to give
            information about a single partition
lp:~wesley-wiedenmeier/curtin/trusty-preserve
    merge into: lp:curtin
    reason: fix handling of disk preservation on
            trusty/precise where blkid may not work
            correctly for virtio disks and some other
            special disks
lp:~wesley-wiedenmeier/curtin/1598310
    merge into: lp:curtin
    reason: (LP: 1598310) The current implementation of
            block.get_blockdev_sector_size fails when
            running on a path for which lsblk may give
            multiple results, such as a disk with
            partitions on it or a dos extended partition.
            This branch fixes this and adds unittests
lp:~wesley-wiedenmeier/curtin/1597522
    merge into: lp:curtin
    reason: (LP: 1597522) A fix that went into trunk a
            while ago as a work around for a bug in
            mkfs.vfat added a '-s' flag to partitioning
            commands, but did so on all filesystem
            commands, not just vfat ones. Some tools,
            like mkfs.ext4 silently ignored it, but it
            caused other tools, such as mkfs.xfs to fail
            and mkfs.btrfs to try to create a filesystem
            assuming the underlying disk had 1 byte blocks.
            This branch fixes that, and adds it to vmtests

Bugs found by storagetests fixed so far:
  - (LP: 1597522) Curtin passes -s flag to all mkfs cmds,
                  not just vfat
    lp:~wesley-wiedenmeier/curtin/1597522
        Already merged into partial-testing, but not
        in trunk
  - (LP: 1592962) U...

[lp:~wesley-wiedenmeier/curtin/partial-testing]

498. By Wesley Wiedenmeier on 2016-07-24

Remove test_cciss from storagetest_runner, as it isn't properly reproducing the
cciss issue

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:498
https://server-team-jenkins.canonical.com/job/curtin-ci/328/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/352/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/328/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:498
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/33/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/353/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci-ppc64/33/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

499. By Wesley Wiedenmeier on 2016-07-28

Merge from trunk for tox env fix

500. By Wesley Wiedenmeier on 2016-07-29

Fix incorrectly named conf file

501. By Wesley Wiedenmeier on 2016-07-29

Added more challenging test of handling weird lvm names to test_clear_holders

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:501
https://server-team-jenkins.canonical.com/job/curtin-ci/207/
Executed test runs:
    SUCCESS: https://server-team-jenkins.canonical.com/job/curtin-ci/nodes=amd64/207
    SUCCESS: https://server-team-jenkins.canonical.com/job/curtin-ci/nodes=ppc64el/207

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/207/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

502. By Wesley Wiedenmeier on 2016-07-29

Merge from trunk

503. By Wesley Wiedenmeier on 2016-07-29

Fix config file lvm_mult_lvols_on_pvol

504. By Wesley Wiedenmeier on 2016-07-29

Fix disk id for lvm_mult_lvols_on_pvol.yaml

505. By Wesley Wiedenmeier on 2016-07-29

Fixed disk id in expected_holders for mult_lvols_on_pvol.yaml

506. By Wesley Wiedenmeier on 2016-08-01

Merge from trunk

507. By Wesley Wiedenmeier on 2016-08-01

Merge from trunk

508. By Wesley Wiedenmeier on 2016-08-01

Removed ntfs test conf as creating ntfs volume writes ~10G of data and slows
tests down too much

509. By Wesley Wiedenmeier on 2016-08-01

Removed reference to ntfs test conf file in test_format

510. By Wesley Wiedenmeier on 2016-08-01

In test_clear_holders, use block.sys_block_path instead of no longer extant
block_meta.block_find_sysfs_path

511. By Wesley Wiedenmeier on 2016-08-01

Fixed up whole_disk_fat conf file (10G disk was too large to make a fat16
filesystem, so use vfat instead) and enabled fat conf file, as bug now fixed in
trunk

512. By Wesley Wiedenmeier on 2016-08-02

Add lots of 'wipe: superblock'

513. By Wesley Wiedenmeier on 2016-08-02

Merge from trunk

514. By Wesley Wiedenmeier on 2016-08-03

Get list of test deps from curtin.deps in storagetest_runner

515. By Wesley Wiedenmeier on 2016-08-03

Disable logical.yaml as it causes parted bug on advanced format disks

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:515
https://jenkins.ubuntu.com/server/job/curtin-ci/208/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/208
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/208/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/208/rebuild

review: Needs Fixing (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

516. By Wesley Wiedenmeier on 2016-08-03

Merge from trunk to bring in test fix

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:516
https://jenkins.ubuntu.com/server/job/curtin-ci/211/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/211
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/211

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/211/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

517. By Wesley Wiedenmeier on 2016-08-04

Remove tests on Wily as it is no longer supported

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:517
https://jenkins.ubuntu.com/server/job/curtin-ci/215/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/215
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/215

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/215/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

518. By Wesley Wiedenmeier on 2016-08-05

Remove all test references to reiserfs as not supported

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:518
https://jenkins.ubuntu.com/server/job/curtin-ci/218/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/218
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/218

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/218/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

519. By Wesley Wiedenmeier on 2016-08-05

For lvm_mult_lvols_on_pvol.yaml, replace dos extended/logical partitioning with
gpt, as the dos logical partitioning causes problems on advanced format disks
sometimes. This allows lvm handling to be tested even though dos
extended/logical has to be disabled

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:519
https://jenkins.ubuntu.com/server/job/curtin-ci/224/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/224
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/224

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/224/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

520. By Wesley Wiedenmeier on 2016-08-05

Merge from trunk

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:520
https://jenkins.ubuntu.com/server/job/curtin-ci/225/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/225
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/225

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/225/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

521. By Wesley Wiedenmeier on 2016-08-05

Remove accidental duplicate wipe statement in mdadm.yaml

522. By Wesley Wiedenmeier on 2016-08-05

Remove unneeded partitions in mdadm.yaml

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:522
https://jenkins.ubuntu.com/server/job/curtin-ci/226/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/226
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/226

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/226/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

523. By Wesley Wiedenmeier on 2016-08-08

Sleep for a second after running bcache verification to make a kernel panic
when unregistering the bcache device during the start of the next test

524. By Wesley Wiedenmeier on 2016-08-08

Merge from trunk to pull in unittest fixes

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:524
https://jenkins.ubuntu.com/server/job/curtin-ci/229/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/229
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/229

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/229/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

525. By Wesley Wiedenmeier on 2016-08-08

Added a dmcrypt clear_holders test, but disabled

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:525
https://jenkins.ubuntu.com/server/job/curtin-ci/230/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/230
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/230

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/230/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

526. By Wesley Wiedenmeier on 2016-08-08

Fix crypt_basic.yaml

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:526
https://jenkins.ubuntu.com/server/job/curtin-ci/231/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/231
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/231

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/231/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

527. By Wesley Wiedenmeier on 2016-08-09

Add allindata.yaml as a clear_holders test

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:527
https://jenkins.ubuntu.com/server/job/curtin-ci/232/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/232
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/232

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/232/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

528. By Wesley Wiedenmeier on 2016-08-09

Fix expected holders for allindata.yaml

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:528
https://jenkins.ubuntu.com/server/job/curtin-ci/233/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/233
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/233

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/233/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

529. By Wesley Wiedenmeier on 2016-08-09

remove expected_holders for backing to volgroup1 in allindata.yaml as it i snot
always possible to predict which logical partition is on each md device

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:529
https://jenkins.ubuntu.com/server/job/curtin-ci/234/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/234
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/234

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/234/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

530. By Wesley Wiedenmeier on 2016-08-20

Merge from trunk to disable tests for wily as it is EOL

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:530
https://jenkins.ubuntu.com/server/job/curtin-ci/246/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/246
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/246

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/246/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

531. By Wesley Wiedenmeier on 2016-08-30

Merge from trunk

532. By Wesley Wiedenmeier on 2016-08-30

Merge in updates to storagetests.test_clear_holders from
lp:~wesley-wiedenmeier/curtin/clear-holders-storagetests

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:532
https://jenkins.ubuntu.com/server/job/curtin-ci/258/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/258
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/258

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/258/rebuild

review: Approve (continuous-integration)

[lp:~wesley-wiedenmeier/curtin/partial-testing]

533. By Wesley Wiedenmeier on 2016-09-15

Merge from trunk to pull optionally ignoring errors in mdadm.mdadm_assemble

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:533
https://jenkins.ubuntu.com/server/job/curtin-ci/260/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=amd64/260
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=ppc64el/260
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=s390x/260

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/260/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I've marked this work in progress.
Ryan and I discussed this some, it is valuable, and we'd like to runt his, but at this point the branch probably needs rework to merge from trunk...
so just categorizing as work-in-progress.

Unmerged revisions

533. By Wesley Wiedenmeier on 2016-09-15

Merge from trunk to pull optionally ignoring errors in mdadm.mdadm_assemble

532. By Wesley Wiedenmeier on 2016-08-30

Merge in updates to storagetests.test_clear_holders from
lp:~wesley-wiedenmeier/curtin/clear-holders-storagetests

531. By Wesley Wiedenmeier on 2016-08-30

Merge from trunk

530. By Wesley Wiedenmeier on 2016-08-20

Merge from trunk to disable tests for wily as it is EOL

529. By Wesley Wiedenmeier on 2016-08-09

remove expected_holders for backing to volgroup1 in allindata.yaml as it i snot
always possible to predict which logical partition is on each md device

528. By Wesley Wiedenmeier on 2016-08-09

Fix expected holders for allindata.yaml

527. By Wesley Wiedenmeier on 2016-08-09

Add allindata.yaml as a clear_holders test

526. By Wesley Wiedenmeier on 2016-08-08

Fix crypt_basic.yaml

525. By Wesley Wiedenmeier on 2016-08-08

Added a dmcrypt clear_holders test, but disabled

524. By Wesley Wiedenmeier on 2016-08-08

Merge from trunk to pull in unittest fixes

=== modified file 'Makefile'
--- Makefile	2016-08-18 16:02:27 +0000
+++ Makefile	2016-09-15 18:06:47 +0000
@@ -39,6 +39,18 @@
 		  echo "   apt-get install -qy python3-sphinx"; exit 1; } 1>&2
 	make -C doc html
 
+storagetests: deps
+	sudo nosetests $(noseopts) tests/storagetests
+
+storagetests3: deps
+	sudo nosetests3 $(noseopts) tests/storagetests
+
+run_storagetests: vmtest-deps
+	nosetests3 $(noseopts) tests/storagetest_runner
+
+deps:
+	sudo bin/curtin -v --install-deps
+
 # By default don't sync images when running all tests.
 vmtest:
 	nosetests3 $(noseopts) tests/vmtests

=== modified file 'curtin/block/__init__.py'
--- curtin/block/__init__.py	2016-08-30 19:21:06 +0000
+++ curtin/block/__init__.py	2016-09-15 18:06:47 +0000
@@ -385,13 +385,21 @@
     return
 
 
-def blkid(devs=None, cache=True):
+def dev_blkid(path, cache=True):
+    """return blkid information for a single device"""
+    blkid_data = blkid([path], cache=cache)
+    if len(blkid_data) == 0:
+        raise ValueError("Did not find blkid info for '%s':" % path)
+    if len(blkid_data) != 1:
+        raise ValueError("blkid '%s' returned multiple results: '%s'" %
+                         path, blkid_data)
+    return next(d for d in blkid_data.values())
+
+
+def blkid(devices=[], cache=True):
     """
     get data about block devices from blkid and convert to dict
     """
-    if devs is None:
-        devs = []
-
     # 14.04 blkid reads undocumented /dev/.blkid.tab
     # man pages mention /run/blkid.tab and /etc/blkid.tab
     if not cache:
@@ -401,6 +409,8 @@
                 os.unlink(cachefile)
 
     cmd = ['blkid', '-o', 'full']
+    cmd.extend(devices)
+
     # blkid output is <device_path>: KEY=VALUE
     # where KEY is TYPE, UUID, PARTUUID, LABEL
     out, err = util.subp(cmd, capture=True)
@@ -626,8 +636,10 @@
 
 
 def sysfs_partition_data(blockdev=None, sysfs_path=None):
-    # given block device or sysfs_path, return a list of tuples
-    # of (kernel_name, number, offset, size)
+    """
+    given block device or sysfs_path, return a list of tuples of
+    (kernel_name, number, offset, size)
+    """
     if blockdev:
         blockdev = os.path.normpath(blockdev)
         sysfs_path = sys_block_path(blockdev)

=== modified file 'curtin/commands/block_meta.py'
--- curtin/commands/block_meta.py	2016-08-11 17:21:40 +0000
+++ curtin/commands/block_meta.py	2016-09-15 18:06:47 +0000
@@ -589,7 +589,8 @@
         # Figure out what point should be
         while len(path) > 0 and path[0] == "/":
             path = path[1:]
-        mount_point = os.path.join(state['target'], path)
+        mount_point = os.path.sep.join([state['target'], path])
+        mount_point = os.path.normpath(mount_point)
 
         # Create mount point if does not exist
         util.ensure_dir(mount_point)

=== modified file 'curtin/commands/curthooks.py'
--- curtin/commands/curthooks.py	2016-08-22 16:20:23 +0000
+++ curtin/commands/curthooks.py	2016-09-15 18:06:47 +0000
@@ -648,7 +648,8 @@
     stack_prefix = state.get('report_stack_prefix', '')
 
     with events.ReportEventStack(
-            name=stack_prefix, reporting_enabled=True, level="INFO",
+            name=stack_prefix + '/writing-config',
+            reporting_enabled=True, level="INFO",
             description="writing config files and configuring apt"):
         write_files(cfg, target)
         do_apt_config(cfg, target)
@@ -669,7 +670,8 @@
                   data=None, target=target)
 
     with events.ReportEventStack(
-            name=stack_prefix, reporting_enabled=True, level="INFO",
+            name=stack_prefix + '/installing-kernel',
+            reporting_enabled=True, level="INFO",
             description="installing kernel"):
         setup_zipl(cfg, target)
         install_kernel(cfg, target)
@@ -678,27 +680,38 @@
         restore_dist_interfaces(cfg, target)
 
     with events.ReportEventStack(
-            name=stack_prefix, reporting_enabled=True, level="INFO",
+            name=stack_prefix + '/setting-up-swap',
+            reporting_enabled=True, level="INFO",
             description="setting up swap"):
         add_swap(cfg, target, state.get('fstab'))
 
     with events.ReportEventStack(
-            name=stack_prefix, reporting_enabled=True, level="INFO",
-            description="apply networking"):
+            name=stack_prefix + '/apply-networking-config',
+            reporting_enabled=True, level="INFO",
+            description="apply networking config"):
         apply_networking(target, state)
 
     with events.ReportEventStack(
-            name=stack_prefix, reporting_enabled=True, level="INFO",
+            name=stack_prefix + '/writing-etc-fstab',
+            reporting_enabled=True, level="INFO",
             description="writing etc/fstab"):
         copy_fstab(state.get('fstab'), target)
 
     with events.ReportEventStack(
-            name=stack_prefix, reporting_enabled=True, level="INFO",
+            name=stack_prefix + '/configuring-multipath',
+            reporting_enabled=True, level="INFO",
             description="configuring multipath"):
         detect_and_handle_multipath(cfg, target)
 
     with events.ReportEventStack(
-            name=stack_prefix, reporting_enabled=True, level="INFO",
+            name=stack_prefix + '/installing-missing-packages',
+            reporting_enabled=True, level="INFO",
+            description="installing missing packages"):
+        install_missing_packages(cfg, target)
+
+    with events.ReportEventStack(
+            name=stack_prefix + '/system-upgrade',
+            reporting_enabled=True, level="INFO",
             description="updating packages on target system"):
         system_upgrade(cfg, target)
 

=== modified file 'curtin/commands/install.py'
--- curtin/commands/install.py	2016-07-13 07:57:01 +0000
+++ curtin/commands/install.py	2016-09-15 18:06:47 +0000
@@ -72,6 +72,17 @@
         pass
 
 
+def copy_install_log(logfile, target, log_target_path):
+    """Copy curtin install log file to target system"""
+    if not logfile:
+        LOG.warn('Cannot copy curtin install log to target, no log exists')
+        return
+
+    LOG.debug('Copying curtin install log to target')
+    target = os.path.sep.join([target, log_target_path])
+    shutil.copy(logfile, os.path.normpath(target))
+
+
 def writeline(fname, output):
     """Write a line to a file."""
     if not output.endswith('\n'):
@@ -421,6 +432,10 @@
         legacy_reporter.report_failure(exp_msg)
         raise e
     finally:
+        log_target_path = instcfg.get('save_install_log',
+                                      '/root/curtin-install.log')
+        if log_target_path:
+            copy_install_log(logfile, workingd.target, log_target_path)
         for d in ('sys', 'dev', 'proc'):
             util.do_umount(os.path.join(workingd.target, d))
         mounted = block.get_mountpoints()

=== modified file 'curtin/reporter/events.py'
--- curtin/reporter/events.py	2016-05-06 17:21:33 +0000
+++ curtin/reporter/events.py	2016-09-15 18:06:47 +0000
@@ -84,6 +84,10 @@
         self.post_files = post_files
         if result not in status:
             raise ValueError("Invalid result: %s" % result)
+        if self.result == status.WARN:
+            self.level = "WARN"
+        elif self.result == status.FAIL:
+            self.level = "ERROR"
 
     def as_string(self):
         return '{0}: {1}: {2}: {3}'.format(
@@ -95,10 +99,6 @@
         data['result'] = self.result
         if self.post_files:
             data['files'] = _collect_file_info(self.post_files)
-        if self.result == status.WARN:
-            data['level'] = "WARN"
-        elif self.result == status.FAIL:
-            data['level'] = "ERROR"
         return data
 
 
@@ -122,10 +122,6 @@
 
     See :py:func:`.report_event` for parameter details.
     """
-    if result == status.SUCCESS:
-        event_description = "finished: " + event_description
-    else:
-        event_description = "failed: " + event_description
     event = FinishReportingEvent(event_name, event_description, result,
                                  post_files=post_files, level=level)
     return report_event(event)
@@ -141,7 +137,6 @@
     :param event_description:
         A human-readable description of the event that has occurred.
     """
-    event_description = "started: " + event_description
     event = ReportingEvent(START_EVENT_TYPE, event_name, event_description,
                            level=level)
     return report_event(event)

=== modified file 'curtin/util.py'
--- curtin/util.py	2016-08-29 17:08:26 +0000
+++ curtin/util.py	2016-09-15 18:06:47 +0000
@@ -195,8 +195,9 @@
                     'Command: %(cmd)s\n'
                     'Exit code: %(exit_code)s\n'
                     'Reason: %(reason)s\n'
-                    'Stdout: %(stdout)r\n'
-                    'Stderr: %(stderr)r')
+                    'Stdout: %(stdout)s\n'
+                    'Stderr: %(stderr)s')
+    stdout_indent_level = 8
 
     def __init__(self, stdout=None, stderr=None,
                  exit_code=None, cmd=None,
@@ -217,14 +218,14 @@
             self.exit_code = exit_code
 
         if not stderr:
-            self.stderr = ''
+            self.stderr = "''"
         else:
-            self.stderr = stderr
+            self.stderr = self._indent_text(stderr)
 
         if not stdout:
-            self.stdout = ''
+            self.stdout = "''"
         else:
-            self.stdout = stdout
+            self.stdout = self._indent_text(stdout)
 
         if reason:
             self.reason = reason
@@ -241,6 +242,11 @@
         }
         IOError.__init__(self, message)
 
+    def _indent_text(self, text):
+        if type(text) == bytes:
+            text = text.decode()
+        return text.replace('\n', '\n' + ' ' * self.stdout_indent_level)
+
 
 class LogTimer(object):
     def __init__(self, logfunc, msg):

=== added file 'doc/devel/README-storagetests.txt'
--- doc/devel/README-storagetests.txt	1970-01-01 00:00:00 +0000
+++ doc/devel/README-storagetests.txt	2016-09-15 18:06:47 +0000
@@ -0,0 +1,191 @@
+== Background ==
+Since there are many possible configurations for curtin disk partitioning, and
+some aspects of disk partitioning such as data preservation and shutting down
+previously existing storage devices cannot be tested with vmtests, a second
+test suite, 'storagetests' is provided to verify these features. The
+storagetests run only the disk partitioning stage of curtin installation, and
+validate that storage configuration was done properly in more detail than the
+vmtests. In order to do this, the storage tests do their validation inside of
+the vm instance they are testing on, and report their results back to the
+storagetest_runner using curtin's reporting module.
+
+== Storage Tests ==
+The storage tests call curtin's block_meta.meta_custom directly, rather than
+using curtin's cli. Since curtin keeps track of its internal state using
+system environment variables, modifications to the system environment are made
+before the tests start. Curtin configuration and calls to block meta functions
+are wrapped by the storagetests.CurtinConfig object. All storage test classes
+inherit from storagetests.BaseStorageTest, which configures test logging and
+reporting. The storage validation code is in classes in storagetests.verifiers
+which are used as mix ins to the test runner classes.
+
+Overview of a storage test running:
+ 1. The storage test class being run inherits from BaseStorageTest as well as
+    individual storage verifiers relevant to the test class.
+ 2. Before any testing starts, a setUpClass defined in BaseStorageTest runs in
+    order to configure logging and reporting. This method looks for the
+    existence of conf file at: storagetests.STORAGE_TEST_REPORTER_CONF_FILE. If
+    this file is present, then it will be loaded and used as configuration for
+    curtin.reporting. If the file is not present, no error will be thrown,
+    allowing the test suite to be used manually outside of storagetest_runner.
+    Logging is set to verbose and a separate log file is created for each test
+    class, starting at the base dir in storagetests.STORAGE_TEST_LOG_DIR.
+ 3. The run() method defined in unittests.TestCase is overridden in
+    BaseStorageTest in order to report success/fail for the test back to
+    storagetest_runner even in the event of an exception while running a test.
+    The actual tests are started by making a call with super() to the real
+    TestCase.run() method.
+ 4. The test runner class has an associated list of test names and
+    configuration files. A base test function in the class is used to run a
+    test on every storage configuration in the list. This is done using
+    nose_parameterized, so that each test instance has a separate test function
+    name.
+ 5. The test configuration files do not have any identifier for the disk to run
+    on present by default, and look for a list of disk id -> path mappings in a
+    json file located at storagetests.STORAGE_TEST_DISK_CONF_FILE. By default
+    the tests use the mapping disk1 -> vdb, disk2 -> vdc...
+ 6. The test runner class calls _config_tester, located in BaseStorageTest,
+    which handles test running. This first checks if the config file for the
+    test has been listed as a disabled test for the test class and skips it if
+    it has. It then runs the configuration once, and if data preservation is to
+    be tested, writes test files into the mountpoints for the configuration.
+    Then, it verifies that storage devices have been configured correctly,
+    running all verification functions present in the test class. Verification
+    functions must have names starting with '_test_'. They start with a leading
+    underscore to ensure that the nosetest suite does not run them on their
+    own. If data preservation is being tested, the config file is then run
+    again, under a new environment, then the mountpoints are checked to ensure
+    that the expected test files are intact.
+ 7. Once the test verification is complete, the BaseStorageTest.run() function
+    dumps the number of errors and failures for the test case into a json file,
+    and the reports back to the storage_test runner if reporting has been
+    configured. In the reported event, the result is set to FAIL if the test
+    did not pass completely, and the log file from test running is posted.
+
+The tests for block_meta.clear_holders() have the same overall structure, but
+instead of testing whether the storage configs were handled correctly, they
+configure storage, verify that the expected devices show up in the
+/sys/block/<name>/holders/ dir for each of the storage config entries,
+then call clear_holders() on each disk that has been configured and ensure that
+the devices have been cleared properly.
+
+== Storage Test Runner ==
+The storage test runner is based on curtin vmtests, and allows all of the
+storage config tests to be run on all supported ubuntu releases and with a
+variety of different disk configurations.
+
+Overview of storagetest_runner:
+ 1. The storagetest_runner uses the same image store as vmtests to retrieve
+    images and query the latest image to use. However, it does not have a
+    mechanism for calling for images to be synced, as this is handled by
+    vmtests already
+ 2. In order to get the test code into the target vm, a tarball is generated of
+    the base directory for the copy of curtin the tests are running off of.
+    A webserver is then set up and run in a separate thread from the main test
+    using the python3 http.server, and a context manager to change cwd to the
+    tmpdir that the curtin tarball has been written into is entered once the vm
+    is started so that the webserver will be able to serve the tarball.
+ 3. In addition to the curtin tarball, the test runner needs to pass reporting
+    configuration and a list of paths to use for test disks into the vm. These
+    test configuration files are written into the directory under the vmtest's
+    tmpdir that is being used as the base for the webserver so that they are
+    available to the vm as well.
+ 4. The test runner then generates cloud-init configuration to install the
+    deps needed to run the storagetests and download the needed files from the
+    httpd. Since these tests need to be able to run on older systems, the
+    cc scripts can be generated either using python3 or python2.
+ 5. While the tests are running, tools.report_webhook_logger.CaptureReporting
+    is used to log all curtin reporting events from the storagetests. Since the
+    tests run block_meta in the same python instance as the test code, the
+    reporting configuration applied by the test runner will apply to block meta
+    as well, so the reporting events received by the test runner need to be
+    filtered for tests sent by the storagetest suite rather than by block_meta.
+    Once the reporting events have been filtered to just the storagetest
+    start/finish events, the finish event results are verified, and the curtin
+    log generated during the test is decoded and written to the log dir of the
+    test runner, allowing inspection if there was a failure.
+ 6. The storagetest_runner only reports success if all of the storage tests run
+    in the vm were either skipped or succeeded, and uses the same rules to
+    decide whether or not to delete the tmpdir it created depending on test
+    results as vmtests.
+
+== Debugging ==
+For the storagetest runner, the same general debugging steps can be used as for
+vmtests, so refer to README-vmtest.txt. For debugging specific storage test
+failures, interactive tests have been provided but disabled for each of the
+storagetest_runner test files. They can be run manually using:
+  'nosetests3 storagetest_runner/test_<name>.py:<Rel><Type><Test>Interactive'
+The interactive tests will boot the same system as would be used for non
+interactive tests, using curses mode for the graphics so that it will work over
+ssh, and do all of the pre configuration needed to start the tests without
+actually starting them. The tests can then be run manually and the output can
+be inspected to determine the cause of failure.
+
+== Running ==
+To run the storagetest_runner on all tests, use:
+  'make run_storagetests'
+
+A single test can be run using:
+  'nosetests3 storagetest_runner/test_<name>.py'
+
+The storagetest_runner should be run as a regular user as it does not sync
+images, which was the only part of vmtests that required root. The
+storagetest_runner test classes can be handled in parallel by nosetests using
+  'nosetests3 --processes=-1 tests/storagetest_runner'
+Note that '-1' can be replaced with how many concurrent processes are desired,
+but will use 1 process per cpu core if left at -1.
+
+Since each vm will have to install several packages before starting, having a
+local repo mirror may help test speed.
+
+To run the actual storagetests, run:
+  'make storagetests' or 'make storagetests3'
+These must be run as root, and will destroy data on the system they run on, so
+they should only be run in a vm or on a test system with care taken to ensure
+that the disks specified for use by storagetests are not important
+
+== Environment Variables ==
+Many of the environment variables for vmtests also work on storagetest_runner,
+but not all. Below are the environment variables which work for
+storagetest_runner.
+
+  * CURTIN_VMTEST_KEEP_DATA_PASS CURTIN_VMTEST_KEEP_DATA_FAIL:
+    default:
+      CURTIN_VMTEST_KEEP_DATA_PASS=none
+      CURTIN_VMTEST_KEEP_DATA_FAIL=all
+    These 2 variables determine what portions of the temporary
+    test data are kept.
+
+    The variables contain a comma ',' delimited list of directories
+    that should be kept in the case of a pass or fail. Additionally,
+    the values 'all' and 'none' are accepted.
+
+    Each storagetest_runner instance that runs has its own sub-directory
+    under the top level CURTIN_VMTEST_TOPDIR. In that directory are
+    directories:
+      boot: files to be served by storagetest_runner internal webserver to
+            configure storagetests
+      install: cloud-init user-data for target image seed disk
+      disks: target disks to be used for tests
+      logs: boot log and storage test logs
+      collect: unused in storagetests
+
+  * CURTIN_VMTEST_TOPDIR: default $TMPDIR/vmtest-<timestamp>
+    vmtest and storagetest_runner put all test data under this value.
+    By default, it creates a directory in TMPDIR (/tmp) named as
+    "vmtest-<timestamp>"
+
+    If you set this value, you must ensure that the directory is either
+    non-existent or clean.
+
+  * CURTIN_VMTEST_LOG: default $TMPDIR/vmtest-<timestamp>.log
+    vmtest and storagetest_runner writes extended log information to this file.
+    The default puts the log along side the TOPDIR.
+
+  * IMAGE_DIR: default /srv/images
+    vmtest image sync (used by storagetest_runner) keeps a mirror of maas
+    ephemeral images in this directory
+
+Environment 'boolean' values:
+  For boolean environment variables the value is considered True
+  if it is any value other than case insensitive 'false', '' or "0"

=== added directory 'examples/storagetests'
=== added file 'examples/storagetests/allindata.yaml'
--- examples/storagetests/allindata.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/allindata.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,222 @@
+showtrace: true
+storage:
+  version: 1
+  config:
+     - id: disk1
+       type: disk
+       ptable: gpt
+       model: QEMU HARDDISK
+       name: main_disk
+       grub_device: 1
+       wipe: superblock
+     - id: bios_boot_partition
+       type: partition
+       size: 1MB
+       device: disk1
+       flag: bios_grub
+       number: 1
+       wipe: superblock
+     - id: sda1
+       type: partition
+       size: 1GB
+       device: disk1
+       number: 2  # XXX: we really need to stop using id with DiskPartnum
+       wipe: superblock
+     - id: sda2
+       type: partition
+       size: 1GB
+       device: disk1
+       number: 3  # XXX: we really need to stop using id with DiskPartnum
+       wipe: superblock
+     - id: sda3
+       type: partition
+       size: 1GB
+       device: disk1
+       number: 4  # XXX: we really need to stop using id with DiskPartnum
+       wipe: superblock
+     - id: sda4
+       type: partition
+       size: 1GB
+       device: disk1
+       number: 5  # XXX: we really need to stop using id with DiskPartnum
+       wipe: superblock
+     - id: sda5
+       type: partition
+       size: 3GB
+       device: disk1
+       number: 6  # XXX: we really need to stop using id with DiskPartnum
+       wipe: superblock
+     - id: disk2
+       type: disk
+       ptable: gpt
+       model: QEMU HARDDISK
+       name: second_disk
+       wipe: superblock
+     - id: sdb1
+       type: partition
+       size: 1GB
+       device: disk2
+       wipe: superblock
+     - id: sdb2
+       type: partition
+       size: 1GB
+       device: disk2
+       wipe: superblock
+     - id: sdb3
+       type: partition
+       size: 1GB
+       device: disk2
+       wipe: superblock
+     - id: sdb4
+       type: partition
+       size: 1GB
+       device: disk2
+       wipe: superblock
+     - id: disk3
+       type: disk
+       ptable: gpt
+       model: QEMU HARDDISK
+       name: third_disk
+       wipe: superblock
+     - id: sdc1
+       type: partition
+       size: 1GB
+       device: disk3
+       wipe: superblock
+     - id: sdc2
+       type: partition
+       size: 1GB
+       device: disk3
+       wipe: superblock
+     - id: sdc3
+       type: partition
+       size: 1GB
+       device: disk3
+       wipe: superblock
+     - id: sdc4
+       type: partition
+       size: 1GB
+       device: disk3
+       wipe: superblock
+     - id: disk4
+       type: disk
+       ptable: gpt
+       model: QEMU HARDDISK
+       name: fourth_disk
+       wipe: superblock
+     - id: sdd1
+       type: partition
+       size: 1GB
+       device: disk4
+       wipe: superblock
+     - id: sdd2
+       type: partition
+       size: 1GB
+       device: disk4
+       wipe: superblock
+     - id: sdd3
+       type: partition
+       size: 1GB
+       device: disk4
+       wipe: superblock
+     - id: sdd4
+       type: partition
+       size: 1GB
+       device: disk4
+       wipe: superblock
+     - id: mddevice0
+       name: md0
+       type: raid
+       raidlevel: 5
+       devices:
+         - sda1
+         - sdb1
+         - sdc1
+       spare_devices:
+         - sdd1
+     - id: mddevice1
+       name: md1
+       type: raid
+       raidlevel: raid6
+       devices:
+         - sda2
+         - sdb2
+         - sdc2
+         - sdd2
+       spare_devices:
+         - sda3
+     - id: mddevice2
+       name: md2
+       type: raid
+       raidlevel: 1
+       devices:
+         - sda4
+         - sdb3
+       spare_devices:
+         - sdc3
+         - sdb4
+     - id: mddevice3
+       name: md3
+       type: raid
+       raidlevel: raid0
+       devices:
+         - sdc4
+         - sdd3
+     - id: volgroup1
+       name: vg1
+       type: lvm_volgroup
+       devices:
+           - mddevice0
+           - mddevice1
+           - mddevice2
+           - mddevice3
+     - id: lvmpart1
+       name: lv1
+       size: 1G
+       type: lvm_partition
+       volgroup: volgroup1
+     - id: lvmpart2
+       name: lv2
+       size: 1G
+       type: lvm_partition
+       volgroup: volgroup1
+     - id: lvmpart3
+       name: lv3
+       type: lvm_partition
+       volgroup: volgroup1
+     - id: dmcrypt0
+       type: dm_crypt
+       volume: lvmpart3
+       key: testkey
+       dm_name: dmcrypt0
+     - id: lv1_fs
+       name: storage
+       type: format
+       fstype: ext3
+       volume: lvmpart1
+     - id: lv2_fs
+       name: storage
+       type: format
+       fstype: ext4
+       volume: lvmpart2
+     - id: dmcrypt_fs
+       name: storage
+       type: format
+       fstype: xfs
+       volume: dmcrypt0
+     - id: sda5_root
+       type: format
+       fstype: ext4
+       volume: sda5
+     - id: sda5_mount
+       type: mount
+       path: /
+       device: sda5_root
+     - id: lv1_mount
+       type: mount
+       path: /srv/data
+       device: lv1_fs
+     - id: lv2_mount
+       type: mount
+       path: /srv/backup
+       device: lv2_fs

=== added file 'examples/storagetests/basicdos.yaml'
--- examples/storagetests/basicdos.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/basicdos.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,63 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: msdos
+        name: main_disk
+        wipe: superblock
+      - id: disk1p1
+        type: partition
+        number: 1
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        number: 2
+        size: 1GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p3
+        type: partition
+        size: 1GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p3
+        type: partition
+        size: 1GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: disk1p2_home
+        type: format
+        fstype: ext4
+        volume: disk1p2
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: disk1p2_mount
+        type: mount
+        path: /home
+        device: disk1p2_home
+      - id: disk2
+        type: disk
+        name: sparedisk
+        wipe: superblock
+      - id: disk3
+        type: disk
+        name: btrfs_volume
+        wipe: superblock
+      - id: btrfs_disk_fmt_id
+        type: format
+        fstype: btrfs
+        volume: disk3
+      - id: btrfs_disk_mnt_id
+        type: mount
+        path: /btrfs
+        device: btrfs_disk_fmt_id

=== added file 'examples/storagetests/bcache_basic.yaml'
--- examples/storagetests/bcache_basic.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/bcache_basic.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,52 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+        grub_device: true
+      - id: disk2
+        type: disk
+        name: cache_disk
+        wipe: superblock
+        ptable: gpt
+      - id: disk1p1
+        type: partition
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        size: 4GB
+        device: disk1
+        wipe: superblock
+      - id: disk2p1
+        type: partition
+        size: 2GB
+        device: disk2
+        wipe: superblock
+      - id: bcache0
+        type: bcache
+        name: cache_one
+        cache_device: disk2p1
+        backing_device: disk1p2
+        cache_mode: writethrough
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: cached_home
+        type: format
+        fstype: ext4
+        volume: bcache0
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: home_mount
+        type: mount
+        path: /home
+        device: cached_home

=== added file 'examples/storagetests/bcache_double.yaml'
--- examples/storagetests/bcache_double.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/bcache_double.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,75 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+        grub_device: true
+      - id: disk2
+        type: disk
+        name: cache_disk
+        wipe: superblock
+        ptable: gpt
+      - id: disk3
+        type: disk
+        wipe: superblock
+        name: second_cache_disk
+      - id: disk1p1
+        type: partition
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        size: 2GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p3
+        type: partition
+        size: 2GB
+        device: disk1
+        wipe: superblock
+      - id: disk2p1
+        type: partition
+        size: 2GB
+        device: disk2
+        wipe: superblock
+      - id: bcache0
+        type: bcache
+        name: cache_one
+        cache_device: disk2p1
+        backing_device: disk1p2
+        cache_mode: writethrough
+      - id: second_bcache
+        type: bcache
+        name: cache_two
+        cache_device: disk3
+        backing_device: disk1p3
+        cache_mode: writeback
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: cached_home
+        type: format
+        fstype: ext4
+        volume: bcache0
+      - id: cached_srv
+        type: format
+        fstype: btrfs
+        volume: second_bcache
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: home_mount
+        type: mount
+        path: /home
+        device: cached_home
+      - id: srv_mount
+        type: mount
+        path: /srv
+        device: cached_srv

=== added file 'examples/storagetests/bcache_shared_cache.yaml'
--- examples/storagetests/bcache_shared_cache.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/bcache_shared_cache.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,71 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+        grub_device: true
+      - id: disk2
+        type: disk
+        name: cache_disk
+        wipe: superblock
+        ptable: gpt
+      - id: disk1p1
+        type: partition
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        size: 4GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p3
+        type: partition
+        size: 2GB
+        device: disk1
+        wipe: superblock
+      - id: disk2p1
+        type: partition
+        size: 2GB
+        device: disk2
+        wipe: superblock
+      - id: bcache0
+        type: bcache
+        name: cache_one
+        cache_device: disk2p1
+        backing_device: disk1p2
+        cache_mode: writethrough
+      - id: bcache1
+        type: bcache
+        name: cache_two
+        cache_device: disk2p1
+        backing_device: disk1p3
+        cache_mode: writeback
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: cached_home
+        type: format
+        fstype: ext4
+        volume: bcache0
+      - id: cached_srv
+        type: format
+        fstype: xfs
+        volume: bcache1
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: home_mount
+        type: mount
+        path: /home
+        device: cached_home
+      - id: srv_mount
+        type: mount
+        path: /srv
+        device: cached_srv

=== added file 'examples/storagetests/crypt_basic.yaml'
--- examples/storagetests/crypt_basic.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/crypt_basic.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,43 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+        grub_device: true
+      - id: disk1p1
+        type: partition
+        number: 1
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        number: 2
+        size: 1GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: crypt_id
+        type: dm_crypt
+        dm_name: crypt0
+        volume: disk1p2
+        key: test_key_123
+      - id: crypt_fmt
+        type: format
+        volume: crypt_id
+        fstype: ext4
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: disk1p2_mount
+        type: mount
+        path: /home
+        device: crypt_fmt

=== added file 'examples/storagetests/diskonlydos.yaml'
--- examples/storagetests/diskonlydos.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/diskonlydos.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,8 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: msdos
+        name: second_disk
+        wipe: superblock

=== added file 'examples/storagetests/diskonlygpt.yaml'
--- examples/storagetests/diskonlygpt.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/diskonlygpt.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,8 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock

=== added file 'examples/storagetests/formats_on_lvm.yaml'
--- examples/storagetests/formats_on_lvm.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/formats_on_lvm.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,67 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        name: main_disk
+        wipe: superblock
+      - id: disk2
+        type: disk
+        name: second_disk
+        wipe: superblock
+      - id: disk3
+        type: disk
+        name: third_disk
+        wipe: superblock
+      - id: volgroup1
+        name: vg1
+        type: lvm_volgroup
+        devices:
+            - disk1
+            - disk2
+            - disk3
+      - id: lvol1
+        type: lvm_partition
+        name: lv1
+        size: 1G
+        volgroup: volgroup1
+      - id: lvol2
+        type: lvm_partition
+        name: lv2
+        size: 1G
+        volgroup: volgroup1
+      - id: lvol3
+        type: lvm_partition
+        name: lv3
+        size: 1G
+        volgroup: volgroup1
+      - id: lvol4
+        type: lvm_partition
+        name: lv4
+        size: 1G
+        volgroup: volgroup1
+      - id: lvol5
+        type: lvm_partition
+        name: lv5
+        size: 1G
+        volgroup: volgroup1
+      - id: vfat_on_lvm
+        volume: lvol1
+        type: format
+        fstype: vfat
+      - id: ext_on_lvm
+        volume: lvol2
+        type: format
+        fstype: ext4
+      - id: btrfs_on_lvm
+        volume: lvol3
+        type: format
+        fstype: btrfs
+      - id: xfs_on_lvm
+        volume: lvol4
+        type: format
+        fstype: xfs
+      - id: swap_on_lvm
+        volume: lvol5
+        type: format
+        fstype: swap

=== added file 'examples/storagetests/gpt_boot.yaml'
--- examples/storagetests/gpt_boot.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/gpt_boot.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,58 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+        grub_device: true
+      - id: disk1_bios_grub
+        type: partition
+        size: 1MB
+        device: disk1
+        flag: bios_grub
+        wipe: superblock
+      - id: disk1p1
+        type: partition
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        size: 1GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: disk1p2_home
+        type: format
+        fstype: ext4
+        volume: disk1p2
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: disk1p2_mount
+        type: mount
+        path: /home
+        device: disk1p2_home
+      - id: disk2
+        type: disk
+        name: sparedisk
+        wipe: superblock
+      - id: disk3
+        type: disk
+        name: btrfs_volume
+        wipe: superblock
+      - id: btrfs_disk_fmt_id
+        type: format
+        fstype: btrfs
+        volume: disk3
+      - id: btrfs_disk_mnt_id
+        type: mount
+        path: /btrfs
+        device: btrfs_disk_fmt_id

=== added file 'examples/storagetests/gpt_simple.yaml'
--- examples/storagetests/gpt_simple.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/gpt_simple.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,54 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+        grub_device: true
+      - id: disk1p1
+        type: partition
+        number: 1
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        number: 2
+        size: 1GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: disk1p2_home
+        type: format
+        fstype: ext4
+        volume: disk1p2
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: disk1p2_mount
+        type: mount
+        path: /home
+        device: disk1p2_home
+      - id: disk2
+        type: disk
+        name: sparedisk
+        wipe: superblock
+      - id: disk3
+        type: disk
+        name: btrfs_volume
+        wipe: superblock
+      - id: btrfs_disk_fmt_id
+        type: format
+        fstype: btrfs
+        volume: disk3
+      - id: btrfs_disk_mnt_id
+        type: mount
+        path: /btrfs
+        device: btrfs_disk_fmt_id

=== added file 'examples/storagetests/logical.yaml'
--- examples/storagetests/logical.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/logical.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,84 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: msdos
+        name: main_disk
+        wipe: superblock
+      - id: disk1primary1
+        type: partition
+        size: 3GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1primary2
+        type: partition
+        size: 2GB
+        device: disk1
+        flag: boot
+        wipe: superblock
+      - id: disk1extended
+        type: partition
+        size: 4GB
+        device: disk1
+        flag: extended
+        wipe: superblock
+      - id: disk1logical1
+        type: partition
+        size: 2GB
+        device: disk1
+        flag: logical
+        wipe: superblock
+      - id: disk1logical2
+        type: partition
+        size: 1GB
+        device: disk1
+        flag: logical
+        wipe: superblock
+      - id: disk1logical3
+        type: partition
+        size: 1GB
+        device: disk1
+        flag: logical
+        wipe: superblock
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1primary1
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: disk1p2_home
+        type: format
+        fstype: ext4
+        volume: disk1primary2
+      - id: disk1p2_mount
+        type: mount
+        path: /home
+        device: disk1p2_home
+      - id: disk1l1_fmt
+        type: format
+        fstype: ext4
+        volume: disk1logical1
+      - id: disk1l1_mnt
+        type: mount
+        path: /media/l1
+        device: disk1l1_fmt
+      - id: disk1l2_fmt
+        type: format
+        fstype: ext4
+        volume: disk1logical2
+      - id: disk1l2_mnt
+        type: mount
+        path: /media/l2
+        device: disk1l2_fmt
+      - id: disk1l3_fmt
+        type: format
+        fstype: ext4
+        volume: disk1logical3
+      - id: disk1l3_mnt
+        type: mount
+        path: /media/l3
+        device: disk1l3_fmt

=== added file 'examples/storagetests/lvm.yaml'
--- examples/storagetests/lvm.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/lvm.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,51 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+      - id: disk1p1
+        type: partition
+        size: 3GB
+        device: disk1
+        wipe: superblock
+        flag: boot
+      - id: disk1p2
+        type: partition
+        size: 2G
+        wipe: superblock
+        device: disk1
+      - id: disk1p3
+        type: partition
+        wipe: superblock
+        size: 3G
+        device: disk1
+      - id: volgroup1
+        name: vg1
+        type: lvm_volgroup
+        devices:
+            - disk1p2
+            - disk1p3
+      - id: lvmpart1
+        name: lv1
+        type: lvm_partition
+        volgroup: volgroup1
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: lv1_fs
+        name: storage
+        type: format
+        fstype: fat32
+        volume: lvmpart1
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: lv1_mount
+        type: mount
+        path: /srv/data
+        device: lv1_fs

=== added file 'examples/storagetests/lvm_mult_lvols_on_pvol.yaml'
--- examples/storagetests/lvm_mult_lvols_on_pvol.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/lvm_mult_lvols_on_pvol.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,74 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        wipe: superblock
+        model: QEMU HARDDISK
+        name: main_disk
+      - id: disk1_bios_grub
+        type: partition
+        size: 1MB
+        device: disk1
+        flag: bios_grub
+      - id: disk1p1
+        type: partition
+        size: 3GB
+        device: disk1
+        wipe: superblock
+      - id: disk1p2
+        type: partition
+        wipe: superblock
+        size: 2G
+        device: disk1
+      - id: disk1p3
+        type: partition
+        wipe: superblock
+        size: 3G
+        device: disk1
+      - id: disk2
+        type: disk
+        wipe: superblock
+      - id: volgroup1
+        name: vg-with-dash
+        type: lvm_volgroup
+        devices:
+            - disk1p2
+            - disk1p3
+            - disk2
+      - id: lvmpart1
+        name: lv-name-one
+        size: 1G
+        type: lvm_partition
+        volgroup: volgroup1
+      - id: lvmpart2
+        name: lv2
+        type: lvm_partition
+        volgroup: volgroup1
+      - id: disk1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: lv1_fs
+        name: storage
+        type: format
+        fstype: fat32
+        volume: lvmpart1
+      - id: lv2_fs
+        name: storage
+        type: format
+        fstype: ext3
+        volume: lvmpart2
+      - id: disk1_mount
+        type: mount
+        path: /
+        device: disk1_root
+      - id: lv1_mount
+        type: mount
+        path: /srv/data
+        device: lv1_fs
+      - id: lv2_mount
+        type: mount
+        path: /srv/backup
+        device: lv2_fs

=== added file 'examples/storagetests/lvm_multiple_vg.yaml'
--- examples/storagetests/lvm_multiple_vg.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/lvm_multiple_vg.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,64 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        name: main_disk
+        wipe: superblock
+        ptable: gpt
+      - id: disk1p1
+        type: partition
+        device: disk1
+        number: 1
+        wipe: superblock
+        size: 2G
+      - id: disk1p2
+        type: partition
+        device: disk1
+        number: 2
+        wipe: superblock
+        size: 2G
+      - id: disk2
+        type: disk
+        name: second_disk
+        wipe: superblock
+      - id: disk3
+        type: disk
+        name: third_disk
+        wipe: superblock
+      - id: volgroup1
+        name: vg1
+        type: lvm_volgroup
+        devices:
+            - disk1p1
+            - disk1p2
+      - id: volgroup2
+        name: vg2
+        type: lvm_volgroup
+        devices:
+            - disk2
+            - disk3
+      - id: vg1-lvol1
+        type: lvm_partition
+        name: lv1
+        volgroup: volgroup1
+      - id: vg2-lvol1
+        type: lvm_partition
+        name: lv1
+        volgroup: volgroup2
+      - id: vfat_on_lvm
+        volume: vg1-lvol1
+        type: format
+        fstype: vfat
+      - id: ext_on_lvm
+        volume: vg2-lvol1
+        type: format
+        fstype: ext4
+      - id: mount1
+        type: mount
+        device: vfat_on_lvm
+        path: /srv/test
+      - id: mount2
+        type: mount
+        device: ext_on_lvm
+        path: /srv/test2

=== added file 'examples/storagetests/lvm_with_dash.yaml'
--- examples/storagetests/lvm_with_dash.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/lvm_with_dash.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,50 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        ptable: gpt
+        name: main_disk
+        wipe: superblock
+      - id: disk1p1
+        type: partition
+        size: 3GB
+        device: disk1
+        wipe: superblock
+        flag: boot
+      - id: disk1p2
+        type: partition
+        size: 2G
+        wipe: superblock
+        device: disk1
+      - id: volgroup1
+        name: volgroup-with-dash
+        type: lvm_volgroup
+        devices:
+            - disk1p2
+      - id: lvmpart1
+        name: lvol-with-dash
+        type: lvm_partition
+        size: 1G
+        volgroup: volgroup1
+      - id: lvmpart2
+        name: lvol---many----dashes_underscore+.abc.def
+        type: lvm_partition
+        volgroup: volgroup1
+      - id: disk1p1_root
+        type: format
+        fstype: ext4
+        volume: disk1p1
+      - id: lv1_fs
+        name: storage
+        type: format
+        fstype: fat32
+        volume: lvmpart1
+      - id: disk1p1_mount
+        type: mount
+        path: /
+        device: disk1p1_root
+      - id: lv1_mount
+        type: mount
+        path: /srv/data
+        device: lv1_fs

=== added file 'examples/storagetests/mdadm.yaml'
--- examples/storagetests/mdadm.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/mdadm.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,59 @@
+storage:
+  version: 1
+  config:
+     - id: disk1
+       type: disk
+       ptable: gpt
+       name: main_disk
+       wipe: superblock
+     - id: bios_boot_partition
+       type: partition
+       size: 1MB
+       device: disk1
+       flag: bios_grub
+       wipe: superblock
+     - id: disk1p1
+       type: partition
+       size: 3GB
+       device: disk1
+       wipe: superblock
+     - id: disk1p2
+       type: partition
+       size: 1GB
+       wipe: superblock
+       device: disk1
+     - id: disk1p3
+       type: partition
+       wipe: superblock
+       size: 1GB
+       device: disk1
+     - id: disk1p4
+       type: partition
+       wipe: superblock
+       size: 1GB
+       device: disk1
+     - id: mddevice
+       name: md0
+       type: raid
+       raidlevel: 1
+       devices:
+         - disk1p2
+         - disk1p3
+       spare_devices:
+         - disk1p4
+     - id: disk1p1_root
+       type: format
+       fstype: ext4
+       volume: disk1p1
+     - id: raid_storage
+       type: format
+       fstype: ext4
+       volume: mddevice
+     - id: disk1p1_mount
+       type: mount
+       path: /
+       device: disk1p1_root
+     - id: raid_mount
+       type: mount
+       path: /media/data
+       device: raid_storage

=== added file 'examples/storagetests/mdadm_bcache.yaml'
--- examples/storagetests/mdadm_bcache.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/mdadm_bcache.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,135 @@
+storage:
+  version: 1
+  config:
+     - grub_device: true
+       id: disk1
+       wipe: superblock
+       type: disk
+       ptable: gpt
+       name: main_disk
+     - id: bios_boot_partition
+       type: partition
+       size: 1MB
+       wipe: superblock
+       device: disk1
+       flag: bios_grub
+       number: 1
+     - id: disk1p1
+       type: partition
+       size: 3GB
+       wipe: superblock
+       device: disk1
+       number: 2  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa1
+     - id: disk1p2
+       type: partition
+       size: 1GB
+       wipe: superblock
+       device: disk1
+       number: 3  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa2
+     - id: disk1p3
+       type: partition
+       wipe: superblock
+       size: 1GB
+       device: disk1
+       number: 4  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa3
+     - id: disk1p4
+       type: partition
+       wipe: superblock
+       size: 1GB
+       device: disk1
+       number: 5  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa4
+     - id: disk1p5
+       wipe: superblock
+       type: partition
+       size: 1GB
+       device: disk1
+       number: 6  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa5
+     - id: disk1p6
+       type: partition
+       size: 1GB
+       wipe: superblock
+       device: disk1
+       number: 7  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa6
+     - id: disk2
+       type: disk
+       name: second_disk
+       wipe: superblock
+     - id: disk3
+       wipe: superblock
+       type: disk
+       ptable: gpt
+       name: third_disk
+     - id: disk3p1
+       type: partition
+       size: 3GB
+       device: disk3
+       wipe: superblock
+       uuid: deadbeef-dead-beef-dead-deadbeefaac1
+     - id: mddevice
+       name: md0
+       type: raid
+       raidlevel: 1
+       devices:
+         - disk1p2
+         - disk1p3
+       spare_devices:
+         - disk1p4
+     - id: bcache1_raid
+       type: bcache
+       name: cached_array
+       backing_device: mddevice
+       cache_device: disk1p5
+       cache_mode: writeback
+     - id: bcache_normal
+       type: bcache
+       name: cached_array_2
+       backing_device: disk1p6
+       cache_device: disk1p5
+       cache_mode: writethrough
+     - id: bcachefoo
+       type: bcache
+       name: cached_array_3
+       backing_device: disk3p1
+       cache_device: disk2
+       cache_mode: writearound
+     - id: disk1p1_fs
+       type: format
+       fstype: ext4
+       volume: disk1p1
+       uuid: deadbeef-dead-beef-dead-deadbeeffff1
+     - id: bcache_raid_storage
+       type: format
+       fstype: ext4
+       volume: bcache1_raid
+       uuid: deadbeef-dead-beef-dead-deadbeefcac1
+     - id: bcache_normal_storage
+       type: format
+       fstype: ext4
+       volume: bcache_normal
+       uuid: deadbeef-dead-beef-dead-deadbeefcac2
+     - id: bcachefoo_fulldiskascache_storage
+       type: format
+       fstype: ext4
+       volume: bcachefoo
+     - id: disk1p1_mount
+       type: mount
+       path: /
+       device: disk1p1_fs
+     - id: bcache1_raid_mount
+       type: mount
+       path: /media/data
+       device: bcache_raid_storage
+     - id: bcache0_mount
+       type: mount
+       path: /media/bcache_normal
+       device: bcache_normal_storage
+     - id: disk1p1_non_root_mount
+       type: mount
+       path: /media/bcachefoo_fulldiskascache_storage
+       device: bcachefoo_fulldiskascache_storage

=== added file 'examples/storagetests/mdadm_lvm.yaml'
--- examples/storagetests/mdadm_lvm.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/mdadm_lvm.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,112 @@
+storage:
+  version: 1
+  config:
+     - grub_device: true
+       id: disk1
+       type: disk
+       ptable: gpt
+       name: main_disk
+       wipe: superblock
+     - id: bios_boot_partition
+       type: partition
+       size: 1MB
+       wipe: superblock
+       device: disk1
+       flag: bios_grub
+       number: 1
+     - id: disk1p1
+       type: partition
+       wipe: superblock
+       size: 3GB
+       device: disk1
+       number: 2  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa1
+     - id: disk1p2
+       type: partition
+       size: 1GB
+       wipe: superblock
+       device: disk1
+       number: 3  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa2
+     - id: disk1p3
+       type: partition
+       size: 1GB
+       wipe: superblock
+       device: disk1
+       number: 4  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa3
+     - id: disk1p4
+       type: partition
+       size: 1GB
+       wipe: superblock
+       device: disk1
+       number: 5  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa4
+     - id: disk1p5
+       type: partition
+       size: 1GB
+       device: disk1
+       wipe: superblock
+       number: 6  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa5
+     - id: disk1p6
+       type: partition
+       size: 1GB
+       device: disk1
+       wipe: superblock
+       number: 7  # XXX: we really need to stop using id with DiskPartnum
+       uuid: deadbeef-dead-beef-dead-deadbeefaaa6
+     - id: disk2
+       type: disk
+       name: second_disk
+       wipe: superblock
+     - id: disk3
+       type: disk
+       wipe: superblock
+       ptable: gpt
+       name: third_disk
+     - id: disk3p1
+       type: partition
+       size: 3GB
+       device: disk3
+       wipe: superblock
+       uuid: deadbeef-dead-beef-dead-deadbeefaac1
+     - id: mddevice
+       name: md0
+       type: raid
+       raidlevel: 1
+       devices:
+         - disk1p2
+         - disk1p3
+       spare_devices:
+         - disk1p4
+     - id: volgroup1
+       name: raid_vg_1
+       devices:
+           - mddevice
+           - disk1p6
+           - disk3p1
+           - disk2
+       type: lvm_volgroup
+     - id: lvol1
+       name: storage_dev
+       type: lvm_partition
+       volgroup: volgroup1
+     - id: disk1p1_fs
+       type: format
+       fstype: ext4
+       volume: disk1p1
+       uuid: deadbeef-dead-beef-dead-deadbeeffff1
+     - id: lvol1_fmt
+       type: format
+       fstype: ext4
+       volume: lvol1
+       uuid: deadbeef-dead-beef-dead-deadbeefcac2
+     - id: disk1p1_mount
+       type: mount
+       path: /
+       device: disk1p1_fs
+     - id: lvol1_mount
+       type: mount
+       path: /media/storage
+       device: lvol1_fmt

=== added file 'examples/storagetests/whole_disk_btrfs_xfs.yaml'
--- examples/storagetests/whole_disk_btrfs_xfs.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/whole_disk_btrfs_xfs.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,19 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        name: main_disk
+        wipe: superblock
+      - id: disk2
+        type: disk
+        name: second_disk
+        wipe: superblock
+      - id: disk1_fmt
+        type: format
+        fstype: btrfs
+        volume: disk1
+      - id: disk2_fmt
+        type: format
+        fstype: xfs
+        volume: disk2

=== added file 'examples/storagetests/whole_disk_ext.yaml'
--- examples/storagetests/whole_disk_ext.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/whole_disk_ext.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,27 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        name: main_disk
+        wipe: superblock
+      - id: disk2
+        type: disk
+        name: second_disk
+        wipe: superblock
+      - id: disk3
+        type: disk
+        name: third_disk
+        wipe: superblock
+      - id: disk1_fmt
+        type: format
+        fstype: ext2
+        volume: disk1
+      - id: disk2_fmt
+        type: format
+        fstype: ext3
+        volume: disk2
+      - id: disk3_fmt
+        type: format
+        fstype: ext4
+        volume: disk3

=== added file 'examples/storagetests/whole_disk_fat.yaml'
--- examples/storagetests/whole_disk_fat.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/whole_disk_fat.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,27 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        name: main_disk
+        wipe: superblock
+      - id: disk2
+        type: disk
+        name: second_disk
+        wipe: superblock
+      - id: disk3
+        type: disk
+        name: third_disk
+        wipe: superblock
+      - id: disk1_fmt
+        type: format
+        fstype: fat
+        volume: disk1
+      - id: disk2_fmt
+        type: format
+        fstype: vfat
+        volume: disk2
+      - id: disk3_fmt
+        type: format
+        fstype: fat32
+        volume: disk3

=== added file 'examples/storagetests/whole_disk_swap.yaml'
--- examples/storagetests/whole_disk_swap.yaml	1970-01-01 00:00:00 +0000
+++ examples/storagetests/whole_disk_swap.yaml	2016-09-15 18:06:47 +0000
@@ -0,0 +1,11 @@
+storage:
+    version: 1
+    config:
+      - id: disk1
+        type: disk
+        name: main_disk
+        wipe: superblock
+      - id: disk1_fmt
+        type: format
+        fstype: swap
+        volume: disk1

=== added directory 'tests/storagetest_runner'
=== added file 'tests/storagetest_runner/__init__.py'
--- tests/storagetest_runner/__init__.py	1970-01-01 00:00:00 +0000
+++ tests/storagetest_runner/__init__.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,471 @@
+import base64
+import json
+import os
+import subprocess
+import textwrap
+import threading
+import time
+import unittest
+from http import server
+from curtin import (util, deps)
+import yaml
+
+import vmtests
+from vmtests import helpers
+from curtin.reporter import events
+from storagetests import (STORAGE_TEST_REPORTER_CONF_FILE,
+                          STORAGE_TEST_REPORT_STACK_PREFIX,
+                          STORAGE_TEST_DISK_CONF_FILE)
+
+from tools.report_webhook_logger import CaptureReporting
+
+LOG = vmtests.logger
+
+# location for curtin on target system
+CURTIN_TAR_PATH = '/tmp/curtin.tar.xz'
+CURTIN_EXTRACT_PATH = '/curtin-storagetests'
+
+TEST_DEPS = [package for (executable, package) in deps.REQUIRED_EXECUTABLES]
+
+
+class ServeInBackground(object):
+    """context manager that runs a webserver in a separate thread"""
+
+    def __init__(self):
+        # when given a addr of 0, first available unused used
+        addr = ('', 0)
+        self.httpd_request_handler = server.SimpleHTTPRequestHandler
+        self.httpd_request_handler.log_request = self.dont_log_request
+        self.httpd = server.HTTPServer(addr, self.httpd_request_handler)
+        self.httpd.server_activate()
+        self.port = self.httpd.server_port
+        self.worker = threading.Thread(target=self.httpd.serve_forever)
+
+    def dont_log_request(self, code=None, size=None):
+        """
+        is able to reqplace SimpleHTTPRequestHandler.log_request to prevent
+        httpd requests from being logged to console
+        """
+        return
+
+    def __enter__(self):
+        self.worker.start()
+        return self
+
+    def __exit__(self, etype, value, trace):
+        self.httpd.shutdown()
+        self.httpd.server_close()
+
+
+class InDirectory(object):
+    """context manager to switch directory while tests are running"""
+
+    def __init__(self, target):
+        self.target = target
+        self.old = os.getcwd()
+
+    def __enter__(self):
+        os.chdir(self.target)
+
+    def __exit__(self, etype, value, trace):
+        os.chdir(self.old)
+
+
+def gen_user_data(local_ip, httpd_port, files_to_retrieve, extra_scripts,
+                  test_py_ver, shutdown, run_tests):
+    """
+    generate user data to download curtin tarball, extract it, run storagetests
+    and report back
+
+    @param local_ip: local ip of test system to be used when accessing local
+                     http servers
+
+    @param httpd_port: port of local httpd for serving config files
+
+    @param files_to_retrieve: a list of files to retrieve and write before
+                              starting tests. each entry is a tuple, item 0 is
+                              the path to the file from the httpd document
+                              root, item 1 is the path to write the file to in
+                              the target
+
+    @param extra_scripts: extra scripts to run before tests
+
+    @param test_py_ver: integer major python version number (either 2 or 3)
+
+    @param shutdown: boolean value to specify if system should be shut down
+                     after tests have been run
+
+    @param run_tests: acutally run the tests
+    """
+
+    if test_py_ver == 2:
+        python_packages = ['python', 'python-pip', 'python-nose']
+        pip_name = 'pip'
+        nosetests = 'nosetests'
+    elif test_py_ver == 3:
+        python_packages = ['python3', 'python3-pip', 'python3-nose']
+        pip_name = 'pip3'
+        nosetests = 'nosetests3'
+    else:
+        raise ValueError("not a valid python major version number: %s",
+                         test_py_ver)
+
+    base_cloudconfig = {
+        'password': 'passw0rd',
+        'chpasswd': {'expire': False},
+        'power_state': {'mode': 'poweroff' if shutdown else 'none'},
+        'network': {'config': 'disabled'},
+    }
+    (ssh_keys, _) = util.subp(['tools/ssh-keys-list', 'cloud-config'],
+                              capture=True)
+
+    # precises' cloud-init version has limited support for
+    # cloud-config-archive and expects cloud-config pieces to be appendable
+    # to a single file and yaml.load()'able.  Resolve this by using
+    # yaml.dump() when generating a list of parts
+    #
+    # precise also does not seem to follow write-files or packages config
+    # entries, so these are handled instead by scripts below
+    parts = [{'type': 'text/cloud-config', 'content': yaml.dump(p, indent=1)}
+             for p in [base_cloudconfig, ssh_keys]]
+
+    # install packages using --no-install-recommends, as mdadm causes postfix
+    # to be installed and that takes quite a bit of extra time
+    install_packages = textwrap.dedent(
+        """
+        apt-get update
+        apt-get install --yes --no-install-recommends {}
+        """).format(' '.join(TEST_DEPS + python_packages))
+
+    # failsafe poweroff runs on precise only, where power_state does not exist
+    precise_poweroff = textwrap.dedent(
+        """
+        [ "$(lsb_release -sc)" = "precise" ] || exit 0;
+        shutdown -P now "Shutting down on precise"
+        """)
+
+    # download files_to_retrieve elements and write
+    download_scripts = []
+    for (httpd_path, target_path) in files_to_retrieve:
+        file_url = ('http://' + local_ip + ':{port:d}/{path}'
+                    .format(port=httpd_port, path=httpd_path))
+        download_scripts.append(textwrap.dedent(
+            """
+            wget -q -O "{target}" "{url}"
+            """).format(target=target_path, url=file_url))
+
+    # extract curtin tarball
+    extract_curtin_tar = textwrap.dedent(
+        """
+        mkdir -p "{extract_loc}"
+        tar xf "{tar_loc}" -C "{extract_loc}"
+        """).format(tar_loc=CURTIN_TAR_PATH, extract_loc=CURTIN_EXTRACT_PATH)
+
+    # install python dependencies via pip
+    python_deps = ['pyyaml', 'nose-parameterized']
+    get_python_deps = "{pip} install {pydeps}".format(
+        pip=pip_name, pydeps=' '.join(python_deps))
+
+    # launch storagetests
+    launch_storagetests = textwrap.dedent(
+        """
+        {target_runner} -w "{extract_loc}" tests/storagetests
+        """).format(extract_loc=CURTIN_EXTRACT_PATH,
+                    target_runner=nosetests)
+
+    scripts = download_scripts + [install_packages, extract_curtin_tar,
+                                  get_python_deps] + extra_scripts
+    if run_tests:
+        scripts.append(launch_storagetests)
+
+    if shutdown:
+        scripts.append(precise_poweroff)
+
+    for part in scripts:
+        if not part.startswith("#!"):
+            part = "#!/bin/sh -x\n" + part
+        LOG.debug('Cloud config archive content (pre-json): %s', part)
+        parts.append({'content': part, 'type': 'text/x-shellscript'})
+
+    return '#cloud-config-archive\n' + json.dumps(parts, indent=1)
+
+
+class TestStorageBase(unittest.TestCase):
+    """
+    Base for storagetest_runner test classes.
+    Handles booting target vm and verifying results
+    """
+    __test__ = False
+    interactive = False
+
+    # appended to scripts to run before starting tests
+    cc_extra_scripts = []
+
+    # major release only
+    test_py_ver = 3
+
+    # should be overridden by relbase mixin
+    arch = None
+    release = None
+    krel = None
+
+    # these probably don't need to be modified
+    boot_timeout = 2400
+    disk_block_size = 512
+
+    # this can be overridden to change disk preferences
+    extra_disks = [('10G', None), ('10G', None), ('10G', None), ('10G', None)]
+
+    # this tells storage tests which disks in target to test on, is merged into
+    # storage config, so can be used to specify disk by any means (path/serial)
+    storage_test_disks = {'disk1': {'path': '/dev/vdb'},
+                          'disk2': {'path': '/dev/vdc'},
+                          'disk3': {'path': '/dev/vdd'},
+                          'disk4': {'path': '/dev/vde'}}
+
+    @classmethod
+    def setUpClass(cls):
+        """setUpClass that does only a single boot with xkvm"""
+
+        local_ip = vmtests.get_lan_ip()
+        setup_start = time.time()
+        LOG.info('Starting setup for testclass: %s', cls.__name__)
+
+        # set up image store with sync disabled, as it is done by vmtests
+        img_store = vmtests.ImageStore(vmtests.IMAGE_SRC_URL,
+                                       vmtests.IMAGE_DIR)
+        img_store.sync = False
+        (img_verstr, (boot_img, boot_kernel, boot_initrd, _)) = \
+            img_store.get_image(cls.release, cls.arch, cls.krel)
+        LOG.debug('Image %s\n boot=%s\n kernel=%s\n initrd=%s\n',
+                  img_verstr, boot_img, boot_kernel, boot_initrd)
+
+        # set up tmpdir, no userdata is passed in yet
+        cls.td = vmtests.TempDir(cls.__name__, "")
+        LOG.info('Using tempdir: %s', cls.td.tmpdir)
+        cls.boot_log = os.path.join(cls.td.logs, 'boot-serial.log')
+        cls.storage_log = os.path.join(cls.td.logs, 'storage-events.json')
+        xout_path = os.path.join(cls.td.logs, 'boot-xkvm.out')
+        LOG.info('Boot console log: %s', cls.boot_log)
+        LOG.info('Storage Test Events: %s', cls.storage_log)
+        httpd_base_dir = cls.td.boot
+
+        # prepare backgrounded webservers
+        storage_events_logger = CaptureReporting(cls.storage_log)
+        publish_srv = ServeInBackground()
+        in_dir = InDirectory(httpd_base_dir)
+
+        # get userdata
+        reporting_conf_path = 'storagetest-reporting.json'
+        test_disk_conf_path = 'storagetest-disk-conf.json'
+        curtin_tar_path = 'curtin.tar.xz'
+        files_to_retrieve = [
+            (reporting_conf_path, STORAGE_TEST_REPORTER_CONF_FILE),
+            (test_disk_conf_path, STORAGE_TEST_DISK_CONF_FILE),
+            (curtin_tar_path, CURTIN_TAR_PATH),
+        ]
+        user_data = gen_user_data(local_ip, publish_srv.port,
+                                  files_to_retrieve, cls.cc_extra_scripts,
+                                  cls.test_py_ver, (not cls.interactive),
+                                  (not cls.interactive))
+        cls.td.write_userdata(user_data)
+
+        # prepare to serve storage test config and curtin tarball
+        reporting_url = ('http://' + local_ip +
+                         ':{:d}/'.format(storage_events_logger.port))
+        reporting_conf = util.json_dumps({
+            'reporting': {
+                'tests': {
+                    'level': 'DEBUG',
+                    'type': 'webhook',
+                    'endpoint': reporting_url,
+                }}})
+        util.write_file(os.path.join(httpd_base_dir, reporting_conf_path),
+                        reporting_conf)
+
+        test_disk_conf = util.json_dumps(cls.storage_test_disks)
+        util.write_file(os.path.join(httpd_base_dir, test_disk_conf_path),
+                        test_disk_conf)
+
+        current_curtin_path = os.getcwd()
+        LOG.info('Building tarball of curtin: %s', current_curtin_path)
+        # tarball is build excluding 'output' as this is where the jenkins test
+        # runner puts the base for the curtin tmp dir and including it causes
+        # tar to try to include the output tarball in itself, making an
+        # infinitely large file
+        subprocess.check_call(
+            ['tar', 'cf', os.path.join(httpd_base_dir, curtin_tar_path),
+             '-C', current_curtin_path, '--exclude=output', '.'],
+            stdout=vmtests.DEVNULL, stderr=subprocess.STDOUT)
+
+        # create boot disk and extra disks
+        block_size_args = ','.join([
+            'logical_block_size={}'.format(cls.disk_block_size),
+            'physical_block_size={}'.format(cls.disk_block_size),
+            'min_io_size={}'.format(cls.disk_block_size), ])
+
+        def format_disk_args(number, src, driver,
+                             fmt=vmtests.TARGET_IMAGE_FORMAT):
+            if driver is None or len(driver) == 0:
+                driver = 'virtio-blk'
+            drive_arg = ','.join(['file={}'.format(src), 'if=none',
+                                  'cache=unsafe', 'format={}'.format(fmt),
+                                  'id=drv{:d}'.format(number),
+                                  'index={:d}'.format(number), ])
+            device_arg = ','.join([driver, 'drive=drv{:d}'.format(number),
+                                   'serial=dev{:d}'.format(number),
+                                   block_size_args, ])
+            return ['-drive', drive_arg, '-device', device_arg]
+
+        disks = []
+        cls.td.boot_disk = os.path.join(cls.td.disks, 'boot_disk.img')
+        subprocess.check_call(['qemu-img', 'create', '-f', 'qcow2',
+                               '-b', boot_img, cls.td.boot_disk, '4G'],
+                              stdout=vmtests.DEVNULL, stderr=subprocess.STDOUT)
+        disks.extend(format_disk_args(0, cls.td.boot_disk, None, fmt='qcow2'))
+
+        for (disk_no, (disk_sz, disk_driver)) in enumerate(cls.extra_disks):
+            dpath = os.path.join(
+                cls.td.disks, 'extra_disk_{:d}.img'.format(disk_no))
+            subprocess.check_call(
+                ['qemu-img', 'create', '-f', vmtests.TARGET_IMAGE_FORMAT,
+                 dpath, disk_sz],
+                stdout=vmtests.DEVNULL,
+                stderr=subprocess.STDOUT)
+            disks.extend(format_disk_args(disk_no + 1, dpath, disk_driver))
+
+        # create xkvm command
+        xkvm = os.path.join(current_curtin_path, 'tools/xkvm')
+        cmd = ([xkvm, '-v', '--no-dowait' if cls.interactive else '--dowait',
+                '--netdev={}'.format(vmtests.DEFAULT_BRIDGE), '--'] + disks +
+               ['-m', '1024',
+                '-curses' if cls.interactive else '-nographic',
+                '-serial', 'file:{}'.format(cls.boot_log)] +
+               ['-drive',
+                'file={},if=virtio,media=cdrom'.format(cls.td.seed_disk)] +
+               ['-kernel', boot_kernel, '-initrd', boot_initrd,
+                '-append', 'root=/dev/vda console=ttyS0 ds=nocloud'])
+
+        # boot storage test runner system
+        try:
+            LOG.info('Booting target image + running tests: %s', cls.boot_log)
+            LOG.debug('%s', ' '.join(cmd))
+            with open(xout_path, 'wb') as fpout:
+                with in_dir, publish_srv, storage_events_logger:
+                    cls.boot_system(cmd, console_log=cls.boot_log,
+                                    proc_out=fpout, timeout=cls.boot_timeout,
+                                    purpose='boot',
+                                    interactive=cls.interactive)
+        except Exception as error:
+            LOG.error('Booting storage test system failed: %s', error)
+            cls.tearDownClass()
+            raise error
+        finally:
+            if os.path.exists(cls.boot_log):
+                content = (util.load_file(cls.boot_log, mode='rb')
+                           .decode('utf-8', errors='replace'))
+                LOG.debug('boot serial console output:\n%s', content)
+            else:
+                LOG.warning('booting test system did not produce console log')
+
+        LOG.info(
+            '{} setUpClass finished. took {:02f} seconds. Running testcases.'
+            .format(cls.__name__, time.time() - setup_start))
+
+    @classmethod
+    def tearDownClass(cls):
+        """determine tests passed and clear tmpdir if needed"""
+
+        success = False
+        sfile = os.path.exists(cls.td.success_file)
+        efile = os.path.exists(cls.td.errors_file)
+        if not (sfile or efile):
+            LOG.warning('class %s had no status. possibly no tests run',
+                        cls.__name__)
+        elif sfile and efile:
+            LOG.warning('class %s had success and fail', cls.__name__)
+        elif sfile:
+            success = True
+
+        vmtests.clean_working_dir(cls.td.tmpdir, success,
+                                  keep_pass=vmtests.KEEP_DATA['pass'],
+                                  keep_fail=vmtests.KEEP_DATA['fail'])
+
+    @classmethod
+    def boot_system(cls, cmd, console_log, proc_out, timeout, purpose,
+                    interactive=False):
+        """boot target, wrapping boot cmd in timout check"""
+
+        def boot_interactive():
+            helpers.check_call(cmd, timeout=timeout)
+            return True
+
+        def myboot():
+            helpers.check_call(cmd, timeout=timeout, stdout=proc_out,
+                               stderr=subprocess.STDOUT)
+            return True
+
+        return vmtests.boot_log_wrap(
+            cls.__name__, boot_interactive if interactive else myboot,
+            cmd, console_log, timeout, purpose)
+
+    def test_reported_results(self):
+        """parsing storage test reporting log to verify test results"""
+        LOG.info('Parsing storage test reporting log to verify test results')
+
+        # load data
+        self.assertTrue(os.path.isfile(self.storage_log))
+        data = json.loads(util.load_file(self.storage_log))
+
+        # filter out events that come from curtin block_meta
+        # since storagetests.make_command_environment sets the report stack
+        # prefix to a known value, all reporting events that come from within
+        # block meta will start with this prefix, while events from
+        # BaseStorageTest.run() will not
+        data = [e for e in data if 'name' in e and not
+                e.get('name').startswith(STORAGE_TEST_REPORT_STACK_PREFIX)]
+
+        # data must be even length because everything needs to have start and
+        # stop
+        self.assertEqual(
+            len(data) % 2, 0,
+            'some reporting data missing, maybe not all tests ran?')
+
+        while len(data) > 0:
+            (start, finish) = data[:2]
+            data = data[2:]
+            self.assertEqual(start.get('event_type'),
+                             events.START_EVENT_TYPE)
+            self.assertEqual(finish.get('event_type'),
+                             events.FINISH_EVENT_TYPE)
+
+            # decode returned curtin log
+            self.assertEqual(len(finish.get('files')), 2)
+            for (enc_log, log_dir_prefix) in zip(
+                    finish.get('files'), ('curtin_logs', 'test_stats')):
+                log_file_basename = os.path.basename(enc_log.get('path'))
+                log_target_path = os.path.join(self.td.logs, log_dir_prefix,
+                                               log_file_basename)
+                decoded = base64.b64decode(enc_log.get('content'))
+                util.write_file(log_target_path, decoded, omode='wb')
+
+            # log result
+            test_result = finish.get('result') == events.status.SUCCESS
+            record_file = (self.td.success_file if test_result
+                           else self.td.errors_file)
+
+            record_listing = []
+            if os.path.exists(record_file):
+                contents = util.load_file(record_file)
+                if len(contents) != 0:
+                    record_listing = json.loads(contents)
+            record_entry = {'runner_class': type(self).__name__,
+                            'test_name': start.get('name'),
+                            'result': finish.get('result')}
+            LOG.debug('test result: %s', record_entry)
+            record_listing.append(record_entry)
+            util.write_file(
+                record_file, util.json_dumps(record_listing))
+
+            self.assertTrue(
+                test_result, 'failed test: {}'.format(start.get('name')))

=== added file 'tests/storagetest_runner/test_advanced_format.py'
--- tests/storagetest_runner/test_advanced_format.py	1970-01-01 00:00:00 +0000
+++ tests/storagetest_runner/test_advanced_format.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,43 @@
+from . import TestStorageBase
+from vmtests.releases import base_vm_classes as relbase
+
+
+class AdvancedFormatTestBase(TestStorageBase):
+    __test__ = False
+    disk_block_size = 4096
+
+
+class PreciseAdvancedFormatTestStorage(relbase.precise_hwe_t,
+                                       AdvancedFormatTestBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class TrustyAdvancedFormatTestStorage(relbase.trusty,
+                                      AdvancedFormatTestBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class XenialAdvancedFormatTestStorage(relbase.xenial,
+                                      AdvancedFormatTestBase):
+    __test__ = True
+
+
+class YakketyAdvancedFormatTestStorage(relbase.yakkety,
+                                       AdvancedFormatTestBase):
+    __test__ = True
+
+
+class TrustyAdvancedFormatTestInteractive(relbase.trusty,
+                                          AdvancedFormatTestBase):
+    # this is just for manually running storagetests, should be turned off
+    __test__ = False
+    interactive = True
+    test_py_ver = 2
+
+
+class XenialAdvancedFormatTestInteractive(relbase.xenial,
+                                          AdvancedFormatTestBase):
+    __test__ = False
+    interactive = True

=== added file 'tests/storagetest_runner/test_basic.py'
--- tests/storagetest_runner/test_basic.py	1970-01-01 00:00:00 +0000
+++ tests/storagetest_runner/test_basic.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,38 @@
+from . import TestStorageBase
+from vmtests.releases import base_vm_classes as relbase
+
+
+class PreciseTestStorage(relbase.precise_hwe_t, TestStorageBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class TrustyTestStorage(relbase.trusty, TestStorageBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class XenialTestStorage(relbase.xenial, TestStorageBase):
+    __test__ = True
+
+
+class YakketyTestStorage(relbase.yakkety, TestStorageBase):
+    __test__ = True
+
+
+class PreciseTestInteractive(relbase.precise_hwe_t, TestStorageBase):
+    # this is just for manual testing and should be left disabled
+    __test__ = False
+    interactive = True
+    test_py_ver = 2
+
+
+class TrustyTestInteractive(relbase.trusty, TestStorageBase):
+    __test__ = False
+    interactive = True
+    test_py_ver = 2
+
+
+class YakketyTestInteractive(relbase.yakkety, TestStorageBase):
+    __test__ = False
+    interactive = True

=== added file 'tests/storagetest_runner/test_nvme.py'
--- tests/storagetest_runner/test_nvme.py	1970-01-01 00:00:00 +0000
+++ tests/storagetest_runner/test_nvme.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,35 @@
+from . import TestStorageBase
+from vmtests.releases import base_vm_classes as relbase
+
+
+class NVMETestBase(TestStorageBase):
+    __test__ = False
+    extra_disks = [('10G', 'nvme'), ('10G', 'nvme'), ('10G', 'nvme')]
+    storage_test_disks = {'disk1': {'path': '/dev/nvme0n1'},
+                          'disk2': {'path': '/dev/nvme1n1'},
+                          'disk3': {'path': '/dev/nvme2n1'},
+                          'disk4': {'path': '/dev/nvmd3n1'}}
+
+
+class PreciseNVMETestStorage(relbase.precise_hwe_t, NVMETestBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class TrustyNVMETestStorage(relbase.trusty, NVMETestBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class XenialNVMETestStorage(relbase.xenial, NVMETestBase):
+    __test__ = True
+
+
+class YakketyNVMETestStorage(relbase.yakkety, NVMETestBase):
+    __test__ = True
+
+
+class YakketyNVMETestInteractive(relbase.yakkety, NVMETestBase):
+    # this is just for manual testing, should be left disabled
+    __test__ = False
+    interactive = True

=== added file 'tests/storagetest_runner/test_scsi.py'
--- tests/storagetest_runner/test_scsi.py	1970-01-01 00:00:00 +0000
+++ tests/storagetest_runner/test_scsi.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,36 @@
+from . import TestStorageBase
+from vmtests.releases import base_vm_classes as relbase
+
+
+class SCSITestBase(TestStorageBase):
+    __test__ = False
+    extra_disks = [('10G', 'scsi-hd'), ('10G', 'scsi-hd'), ('10G', 'scsi-hd'),
+                   ('10G', 'scsi-hd')]
+    storage_test_disks = {'disk1': {'path': '/dev/sda'},
+                          'disk2': {'path': '/dev/sdb'},
+                          'disk3': {'path': '/dev/sdc'},
+                          'disk4': {'path': '/dev/sdd'}}
+
+
+class PreciseSCSITestStorage(relbase.precise_hwe_t, SCSITestBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class TrustySCSITestStorage(relbase.trusty, SCSITestBase):
+    __test__ = True
+    test_py_ver = 2
+
+
+class XenialSCSITestStorage(relbase.xenial, SCSITestBase):
+    __test__ = True
+
+
+class YakketySCSITestStorage(relbase.yakkety, SCSITestBase):
+    __test__ = True
+
+
+class YakketySCSITestInteractive(relbase.yakkety, SCSITestBase):
+    # this is just for manual testing, should be left disabled
+    __test__ = False
+    interactive = True

=== added directory 'tests/storagetests'
=== added file 'tests/storagetests/__init__.py'
--- tests/storagetests/__init__.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/__init__.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,261 @@
+import json
+import logging
+import os
+import shutil
+import tempfile
+import unittest
+
+from collections import namedtuple
+from collections import OrderedDict
+
+from curtin import (block, util, config, reporter)
+from curtin.commands import block_meta
+from curtin.reporter import events
+
+STORAGE_TEST_REPORTER_CONF_FILE = '/storagetests-reporting.json'
+STORAGE_TEST_REPORT_STACK_PREFIX = 'curtin/storagetests'
+STORAGE_TEST_LOG_DIR = '/tmp/storagetest_logs'
+STORAGE_TEST_DISK_CONF_FILE = '/storagetests-disks.json'
+
+VERBOSE_LOG_LEVEL = 2
+
+
+def make_command_environment(tdir):
+    """Configure os.environ to match setup during real curtin install"""
+    # copied from util.load_command_environment
+    mapping = {'scratch': 'WORKING_DIR', 'fstab': 'OUTPUT_FSTAB',
+               'interfaces': 'OUTPUT_INTERFACES', 'config': 'CONFIG',
+               'target': 'TARGET_MOUNT_POINT',
+               'network_state': 'OUTPUT_NETWORK_STATE',
+               'network_config': 'OUTPUT_NETWORK_CONFIG',
+               'report_stack_prefix': 'CURTIN_REPORTSTACK'}
+
+    directories = ('WORKING_DIR', 'TARGET_MOUNT_POINT')
+    data = {env_name: os.path.join(tdir, name)
+            for name, env_name in mapping.items()}
+    for directory in directories:
+        util.ensure_dir(data[directory])
+    data['CURTIN_REPORTSTACK'] = STORAGE_TEST_REPORT_STACK_PREFIX
+
+    # set verbosity to high
+    data['CURTIN_VERBOSITY'] = str(VERBOSE_LOG_LEVEL)
+
+    os.environ.update(data)
+
+
+class BaseStorageTest(unittest.TestCase):
+    """
+    Base class for storage test runner classes.
+    Configures logging, reporting, and provides test runner function
+    """
+    __test__ = False
+    disabled_tests = []
+    preserve_test_files = [('test1.txt', 'partition preserve test one'),
+                           ('test2', 'test #2'),
+                           ('test3.txt', 'last test')]
+
+    @classmethod
+    def setUpClass(cls):
+        """set up reporter if reporter configuration available"""
+        cls.report = os.path.exists(STORAGE_TEST_REPORTER_CONF_FILE)
+        cls.report_prefix = cls.__name__
+
+        # set up local log
+        if not os.path.exists(STORAGE_TEST_LOG_DIR):
+            util.ensure_dir(STORAGE_TEST_LOG_DIR)
+        log_file_name = '{}.log'.format(cls.report_prefix)
+        report_dump_name = '{}_results.json'.format(cls.report_prefix)
+        cls.log_file = os.path.join(STORAGE_TEST_LOG_DIR, log_file_name)
+        cls.report_dump = os.path.join(STORAGE_TEST_LOG_DIR, report_dump_name)
+        cls.log = logging.getLogger()
+        handler = logging.FileHandler(filename=cls.log_file)
+        handler.setLevel(VERBOSE_LOG_LEVEL)
+        cls.log.addHandler(handler)
+        cls.log.setLevel(VERBOSE_LOG_LEVEL)
+
+        # init reporter
+        if cls.report:
+            conf = json.loads(util.load_file(STORAGE_TEST_REPORTER_CONF_FILE))
+            reporter.update_configuration(conf.get('reporting', conf))
+
+    def run(self, result=None):
+        """run actual test (overridden from TestCase.run)"""
+        test_reporter = events.ReportEventStack(
+            name=self.id(), description='storage test',
+            reporting_enabled=True, level='INFO',
+            post_files=[self.log_file, self.report_dump])
+        with test_reporter:
+            super(BaseStorageTest, self).run(result)
+            json_result = util.json_dumps({
+                'errors': [str(e[0]) for e in result.errors],
+                'failures': [str(e[0]) for e in result.failures],
+            })
+            util.write_file(self.report_dump, json_result)
+            if result.test.passed not in (True, None):
+                test_reporter.result = 'FAIL'
+
+    def _get_config(self, conf_file):
+        """
+        check if current config file has been disabled and skip if it has.
+        otherwise load CurtinConfig object with config file and set paths
+        """
+        if conf_file in self.disabled_tests:
+            self.skipTest('testing conf file disabled: {}'.format(conf_file))
+        conf = CurtinConfig(conf_file)
+        if not conf.set_test_disk_paths():
+            self.skipTest('not enough disks for config: {}'.format(conf_file))
+        return conf
+
+    def _get_test_functions(self):
+        """find all functions in current test class staring with _test_"""
+        return [getattr(self, f) for f in dir(self) if f.startswith('_test_')]
+
+    def _config_tester(self, conf_file, with_preserve=False):
+        """run block meta with specified config file and examine results"""
+        conf = self._get_config(conf_file)
+        with CurtinEnvironment() as env:
+            conf.run()
+            for test_function in self._get_test_functions():
+                test_function(conf, env)
+
+            # before leaving test env and removing tmpdir create files for
+            # preserve test
+            if with_preserve:
+                conf.set_preserves()
+                for mountpoint in conf.preserved_mountpoints():
+                    path = env.join_mount_path(mountpoint)
+                    for test_file in self.preserve_test_files:
+                        util.write_file(os.path.join(path, test_file[0]),
+                                        test_file[1])
+
+        # check that preserve works
+        if with_preserve:
+            with CurtinEnvironment() as env:
+                conf.run()
+                for mountpoint in conf.preserved_mountpoints():
+                    path = env.join_mount_path(mountpoint)
+                    for test_file in self.preserve_test_files:
+                        test_file_path = os.path.join(path, test_file[0])
+                        self.assertTrue(os.path.isfile(test_file_path))
+                        self.assertEqual(util.load_file(test_file_path),
+                                         test_file[1])
+
+
+class CurtinConfig(object):
+    """
+    class to handle configuration file loading/parsing
+    also wraps calls to block_meta
+    """
+
+    conf_base = "examples/storagetests"
+
+    def __init__(self, conf_file):
+        # read config
+        conf_path = os.path.join(self.conf_base, conf_file)
+        if not os.path.exists(conf_path):
+            raise ValueError("could not find conf file %s" % conf_path)
+        self.config = config.load_config(conf_path)
+
+    @property
+    def actual_config(self):
+        """the 'real' storage configuration (list of config entries)"""
+        return self.config['storage']['config']
+
+    def set_test_disk_paths(self):
+        """set paths for disks in config, return True iff all paths valid"""
+        default = {'disk1': {'path': '/dev/vdb'},
+                   'disk2': {'path': '/dev/vdc'},
+                   'disk3': {'path': '/dev/vdd'}}
+        paths = (json.loads(util.load_file(STORAGE_TEST_DISK_CONF_FILE))
+                 if os.path.exists(STORAGE_TEST_DISK_CONF_FILE) else default)
+        for disk in self.find_matching('type', 'disk'):
+            disk.update(paths.get(disk['id'], {}))
+
+        # only return true if curtin can find a path to every disk
+        return not any(p for p in [self.path_to(d) for d in paths]
+                       if p is None or not block.is_valid_device(p))
+
+    def set_preserves(self):
+        """Set preserve flags everywhere in config that supports them"""
+        preservable_types = ["disk", "partition", "format", "lvm_volgroup",
+                             "lvm_partition"]
+        for preservable in preservable_types:
+            entries = self.find_matching('type', preservable)
+            for entry in entries:
+                entry['preserve'] = True
+
+    def preserved_mountpoints(self):
+        """get list of mountpoints on filesystems marked 'preserve'"""
+        mps = self.find_matching('type', 'mount')
+        return list(mp['path'] for mp in mps
+                    if self.with_id(mp['device']).get('preserve'))
+
+    def find_matching(self, attr, val):
+        """search through config for matching entry"""
+        return list(i for i in self.actual_config if i.get(attr) == val)
+
+    def with_id(self, target):
+        """find config entry with given id"""
+        entry = self.find_matching('id', target)
+        if entry is None or len(entry) != 1:
+            return None
+        return entry[0]
+
+    def path_to(self, target):
+        """
+        wrapper for block_meta.get_path_to_storage_volume that handles
+        failure more cleanly
+        """
+        try:
+            if isinstance(target, dict):
+                target = target.get('id')
+            path = block_meta.get_path_to_storage_volume(
+                target, self.as_ordered_dict())
+        except (ValueError, NotImplementedError):
+            return None
+        return path
+
+    def kname_of(self, target):
+        """get kname for target"""
+        path = self.path_to(target)
+        return os.path.split(path)[-1]
+
+    def as_ordered_dict(self):
+        """
+        get dict that matches block_meta.extract_storage_config_odict
+        """
+        return OrderedDict((d['id'], d) for (i, d) in
+                           enumerate(self.actual_config))
+
+    def run(self):
+        """start block_meta.meta_custom on loaded config"""
+        arg_holder = namedtuple('args', 'config')
+        args = arg_holder(config=self.config)
+        block_meta.meta_custom(args)
+
+
+class CurtinEnvironment(object):
+    """context manager that approximates curtin internal state env vars"""
+
+    def __init__(self):
+        self.tmpdir = tempfile.mkdtemp()
+
+    def __enter__(self):
+        make_command_environment(self.tmpdir)
+        return self
+
+    def join_mount_path(self, abs_path):
+        """get path to mountpoint in tmpdir from target mount path"""
+        while len(abs_path) > 0 and abs_path[0] == "/":
+            abs_path = abs_path[1:]
+        return os.path.join(self.tmpdir, 'target', abs_path)
+
+    def __exit__(self, etype, value, traceback):
+        # Note, this is unmount technique at end of
+        # curtin.commands.install.cmd_install
+        mounted = block.get_mountpoints()
+        mounted.sort(key=lambda x: -1 * x.count("/"))
+        for directory in [d for d in mounted if self.tmpdir in d]:
+            util.do_umount(directory)
+        util.do_umount(os.path.join(self.tmpdir, 'target'))
+        shutil.rmtree(self.tmpdir)

=== added file 'tests/storagetests/test_bcache.py'
--- tests/storagetests/test_bcache.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_bcache.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,17 @@
+from . import BaseStorageTest
+from .verifiers import (BasicTests, BcacheTests)
+from nose_parameterized import (parameterized, param)
+
+BCACHE_TESTS = [
+    param('simple bcache', 'bcache_basic.yaml'),
+    param('bcache with shared cache dev', 'bcache_shared_cache.yaml'),
+    param('two separate caches', 'bcache_double.yaml'),
+]
+
+
+class TestBcache(BaseStorageTest, BasicTests, BcacheTests):
+    __test__ = True
+
+    @parameterized.expand(BCACHE_TESTS)
+    def test_bcache(self, _, conf_file, with_preserve=False):
+        self._config_tester(conf_file, with_preserve=with_preserve)

=== added file 'tests/storagetests/test_clear_holders.py'
--- tests/storagetests/test_clear_holders.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_clear_holders.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,105 @@
+from . import (BaseStorageTest, CurtinEnvironment)
+from curtin.block import clear_holders
+from nose_parameterized import (parameterized, param)
+
+CLEAR_HOLDERS_TESTS = [
+    param('basic gpt', 'gpt_simple.yaml'),
+    param('dos logical', 'logical.yaml'),
+    param('simple raid', 'mdadm.yaml',
+          expected_holders={'disk1p2': {'md0'}, 'disk1p3': {'md0'}}),
+    param('simple lvm', 'lvm.yaml',
+          expected_holders={'disk1p2': {'dm-0'}, 'disk1p3': {'dm-0'}}),
+    param('lvm multiple volgroups', 'lvm_multiple_vg.yaml',
+          expected_holders={'disk1p1': {'dm-0'},
+                            'disk1p2': {'dm-0'},
+                            'disk2': {'dm-1'},
+                            'disk3': {'dm-1'}}),
+    param('lvm bug 1592962 dash in name', 'lvm_with_dash.yaml',
+          expected_holders={'disk1p2': {'dm-0', 'dm-1'}}),
+    param('lvm bug 1588875 mult lvols on pvol', 'lvm_mult_lvols_on_pvol.yaml',
+          expected_holders={'disk1p2': {'dm-0', 'dm-1'},
+                            'disk1p3': {'dm-1'},
+                            'disk2': {'dm-1'}}),
+    param('simple bcache', 'bcache_basic.yaml',
+          expected_holders={'disk1p2': {'bcache0'}, 'disk2p1': {'bcache0'}}),
+    param('two separate bcaches', 'bcache_double.yaml',
+          expected_holders={'disk1p2': {'bcache0'},
+                            'disk1p3': {'bcache1'},
+                            'disk2p1': {'bcache0'},
+                            'disk3': {'bcache1'}}),
+    param('shared cache device bcache', 'bcache_shared_cache.yaml',
+          expected_holders={'disk1p2': {'bcache0'},
+                            'disk1p3': {'bcache1'},
+                            'disk2p1': {'bcache0', 'bcache1'}}),
+    param('bcache on mdadm', 'mdadm_bcache.yaml',
+          expected_holders={'disk1p2': {'md0'},
+                            'disk1p3': {'md0'},
+                            'disk1p4': {'md0'},
+                            'disk1p5': {'bcache0', 'bcache1'},
+                            'disk1p6': {'bcache1'},
+                            'disk2': {'bcache2'},
+                            'disk3p1': {'bcache2'},
+                            'md0': {'bcache0'}}),
+    param('simple cryptseup', 'crypt_basic.yaml',
+          expected_holders={'disk1p2': {'dm-0'}}),
+    param('all in data', 'allindata.yaml',
+          expected_holders={'sda1': {'md0'}, 'sdb1': {'md0'}, 'sdc1': {'md0'},
+                            'sdd1': {'md0'}, 'sda2': {'md1'}, 'sdb2': {'md1'},
+                            'sdc2': {'md1'}, 'sdd2': {'md1'}, 'sda3': {'md1'},
+                            'sda4': {'md2'}, 'sdb3': {'md2'}, 'sdc3': {'md2'},
+                            'sdb4': {'md2'}, 'sdc4': {'md3'}, 'sdd3': {'md3'},
+                            'lvmpart3': {'dm-3'}}),
+]
+
+
+class TestClearHolders(BaseStorageTest):
+    __test__ = True
+    # there is a higher chance of the runner vm encountering a kernel panic
+    # when running mdadm_bcache.yaml than any of the other bcache tests, so it
+    # cannot be enabled here
+
+    # mdadm.yaml cannot be enabled because when running many tests in parallel
+    # it sometimes takes way too long for the raid array to assemble
+
+    # logical.yaml is disabled because of #1610628
+
+    # allindata.yaml and crypt_basic.yaml are disabled because of #1611452,
+    # which likely will not be fixed. however, clear_holders does work with
+    # these files enabled in all situations where curtin is able to create
+    # crypt devices successfully
+    disabled_tests = ['mdadm_bcache.yaml', 'mdadm.yaml', 'logical.yaml',
+                      'allindata.yaml', 'crypt_basic.yaml']
+
+    @parameterized.expand(CLEAR_HOLDERS_TESTS)
+    def test_clear(self, _, conf_file, expected_holders={}):
+        conf = self._get_config(conf_file)
+        self.log.info('Testing clear_holders on: {}'.format(conf_file))
+
+        # set up test config
+        with CurtinEnvironment():
+            conf.run()
+
+        # check that holders are as expected
+        for (dev_id, holders) in expected_holders.items():
+            self.log.info('verifying dev with id: %s has holders: %s',
+                          dev_id, holders)
+            path = conf.path_to(dev_id)
+            self.assertIsNotNone(path)
+            found = set(clear_holders.get_holders(path))
+            self.assertEqual(found, holders, "Incorrect holders on '{}': {}"
+                             .format(dev_id, holders))
+
+        # run clear_holders on all paths
+        paths = [conf.path_to(e) for e in conf.find_matching('type', 'disk')]
+        clear_holders.start_clear_holders_deps()
+        clear_holders.clear_holders(paths)
+
+        # make sure everything is clear manually
+        for path in paths:
+            remaining_holders = clear_holders.get_holders(path)
+            self.assertEqual(len(remaining_holders), 0,
+                             "Clear holders failed, '{}' still held by: {}"
+                             .format(path, remaining_holders))
+
+        # make sure that assert_clear works
+        clear_holders.assert_clear(paths)

=== added file 'tests/storagetests/test_complex.py'
--- tests/storagetests/test_complex.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_complex.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,17 @@
+from . import BaseStorageTest
+from .verifiers import (BasicTests, BcacheTests, LvmTests, RaidTests)
+from nose_parameterized import (parameterized, param)
+
+COMPLEX_TESTS = [
+    param('all in data', 'allindata.yaml')
+]
+
+
+class TestComplexStorage(BaseStorageTest, BasicTests, BcacheTests,
+                         LvmTests, RaidTests):
+    __test__ = True
+    disabled_tests = ['allindata.yaml']
+
+    @parameterized.expand(COMPLEX_TESTS)
+    def test_complex(self, _, conf_file, with_preserve=False):
+        self._config_tester(conf_file, with_preserve=with_preserve)

=== added file 'tests/storagetests/test_disk_partitions.py'
--- tests/storagetests/test_disk_partitions.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_disk_partitions.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,21 @@
+from . import BaseStorageTest
+from .verifiers import (BasicTests)
+from nose_parameterized import (parameterized, param)
+
+BASIC_PARTITIONING_TESTS = [
+    param('dos disk only', 'diskonlydos.yaml'),
+    param('gpt disk only', 'diskonlygpt.yaml'),
+    param('basic gpt', 'gpt_simple.yaml', with_preserve=True),
+    param('gpt with bios_grub', 'gpt_boot.yaml', with_preserve=True),
+    param('basic dos', 'basicdos.yaml', with_preserve=True),
+    param('dos logical', 'logical.yaml', with_preserve=True),
+]
+
+
+class TestBasic(BaseStorageTest, BasicTests):
+    __test__ = True
+    disabled_tests = ['logical.yaml']
+
+    @parameterized.expand(BASIC_PARTITIONING_TESTS)
+    def test_basic_config(self, _, conf_file, with_preserve=False):
+        self._config_tester(conf_file, with_preserve=with_preserve)

=== added file 'tests/storagetests/test_format.py'
--- tests/storagetests/test_format.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_format.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,19 @@
+from . import BaseStorageTest
+from .verifiers import BasicTests
+from nose_parameterized import (parameterized, param)
+
+FORMAT_TESTS = [
+    param('whole disk ext fs', 'whole_disk_ext.yaml'),
+    param('whole disk btrfs xfs', 'whole_disk_btrfs_xfs.yaml'),
+    param('whole disk dos formats', 'whole_disk_fat.yaml'),
+    param('whole disk swap', 'whole_disk_swap.yaml'),
+    param('formats on lvm', 'formats_on_lvm.yaml'),
+]
+
+
+class TestWholeDiskFormat(BaseStorageTest, BasicTests):
+    __test__ = True
+
+    @parameterized.expand(FORMAT_TESTS)
+    def test_whole_disk_formats(self, _, conf_file):
+        self._config_tester(conf_file)

=== added file 'tests/storagetests/test_layers_on_mdadm.py'
--- tests/storagetests/test_layers_on_mdadm.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_layers_on_mdadm.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,21 @@
+from . import BaseStorageTest
+from .verifiers import (BasicTests, RaidTests, BcacheTests, LvmTests)
+from nose_parameterized import (parameterized, param)
+
+LAYERS_ON_MDADM_TESTS = [
+    param('bcache on mdadm', 'mdadm_bcache.yaml'),
+    param('lvm on mdadm', 'mdadm_lvm.yaml'),
+]
+
+
+class TestLayersOnMdadm(BaseStorageTest, BasicTests, RaidTests,
+                        BcacheTests, LvmTests):
+    __test__ = True
+    # about 50% of the time running md_check on these devices fails with:
+    # md_check failed with exception Array syncing, not idle state: /dev/md0
+    # (I think when the next storage layer is created it causes a resync)
+    disabled_tests = ['mdadm_bcache.yaml', 'mdadm_lvm.yaml']
+
+    @parameterized.expand(LAYERS_ON_MDADM_TESTS)
+    def test_layers_on_mdadm(self, _, conf_file, with_preserve=False):
+        self._config_tester(conf_file, with_preserve=with_preserve)

=== added file 'tests/storagetests/test_lvm.py'
--- tests/storagetests/test_lvm.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_lvm.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,16 @@
+from . import BaseStorageTest
+from .verifiers import (BasicTests, LvmTests)
+from nose_parameterized import (parameterized, param)
+
+LVM_TESTS = [
+    param('simple lvm', 'lvm.yaml', with_preserve=True),
+    param('multiple volgroups', 'lvm_multiple_vg.yaml', with_preserve=True),
+]
+
+
+class TestLVM(BaseStorageTest, BasicTests, LvmTests):
+    __test__ = True
+
+    @parameterized.expand(LVM_TESTS)
+    def test_lvm(self, _, conf_file, with_preserve=False):
+        self._config_tester(conf_file, with_preserve=with_preserve)

=== added file 'tests/storagetests/test_raid.py'
--- tests/storagetests/test_raid.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/test_raid.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,19 @@
+from . import BaseStorageTest
+from .verifiers import (BasicTests, RaidTests)
+from nose_parameterized import (parameterized, param)
+
+MDADM_TESTS = [
+    param('simple mdadm', 'mdadm.yaml', with_preserve=True),
+]
+
+
+class TestRaid(BaseStorageTest, BasicTests, RaidTests):
+    __test__ = True
+    # there is a race waiting for the mdadm device to start up and shut down
+    # when the host system is under stress from tests running in parallel
+    # that makes it impossible to enable this file here
+    disabled_tests = ['mdadm.yaml']
+
+    @parameterized.expand(MDADM_TESTS)
+    def test_mdadm(self, _, conf_file, with_preserve=False):
+        self._config_tester(conf_file, with_preserve=with_preserve)

=== added file 'tests/storagetests/verifiers.py'
--- tests/storagetests/verifiers.py	1970-01-01 00:00:00 +0000
+++ tests/storagetests/verifiers.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,222 @@
+import os
+import time
+from curtin import (block, util)
+
+
+class BasicTests(object):
+    """Verifer for disk partitions, formats and mounts"""
+    __test__ = False
+
+    # test functions shared between most configs:
+    def _test_ptable(self, conf, env):
+        """test that the right partition table has been set up"""
+        for conf_entry in conf.find_matching('type', 'disk'):
+            needed_ptable = conf_entry.get('ptable')
+            dev_path = conf.path_to(conf_entry)
+            self.assertIsNotNone(dev_path)
+            self.assertTrue(block.is_valid_device(dev_path))
+            try:
+                current_ptable = block.dev_blkid(
+                    dev_path, cache=False)['PTTYPE']
+            except (util.ProcessExecutionError, KeyError):
+                # blkid did not get info or failed to run, will be fixed when
+                # lp:~wesley-wiedenmeier/curtin/blkid lands
+                continue
+
+            # if ptable not in config then don't need one to be created
+            if needed_ptable is None:
+                continue
+
+            if needed_ptable == 'gpt':
+                self.assertEqual(current_ptable, 'gpt')
+            elif needed_ptable in ['dos', 'msdos']:
+                self.assertIn(current_ptable, ['dos', 'msdos'])
+
+    def _test_partitions(self, conf, env):
+        """make sure all required partitions exist and are the right size"""
+        for conf_entry in conf.find_matching('type', 'partition'):
+            # Do not try to verify size of extended partitions, as releases
+            # after trusty will report 0 in /sys/class/block/*/size for the
+            # extended partition
+            if conf_entry.get('flag') == 'extended':
+                continue
+
+            # check it was created
+            dev_path = conf.path_to(conf_entry)
+            self.assertIsNotNone(dev_path)
+            self.assertTrue(block.is_valid_device(dev_path))
+
+            # check size
+            # if none specified, dont try to check
+            if conf_entry.get('size') is None:
+                continue
+
+            sysfs_size_path = os.path.join(
+                '/sys/class/block', conf.kname_of(conf_entry), 'size')
+            part_size_blocks = int(util.load_file(sysfs_size_path))
+            # sysfs size is always reported in 512 byte blocks even if the
+            # underlying disk is advanced format, so do not load
+            # queue/logical_blocks_size to get multiplier
+            part_size = 512 * part_size_blocks
+            req_size = util.human2bytes(conf_entry.get('size'))
+            self.assertEqual(req_size, part_size)
+
+    def _test_format(self, conf, env):
+        """make sure formats created properly"""
+        for conf_entry in conf.find_matching('type', 'format'):
+            req_fs = conf_entry.get('fstype')
+            volume_conf = conf.with_id(conf_entry.get('volume'))
+
+            # get current fs
+            lsblk_info = block._lsblock().get(conf.kname_of(volume_conf))
+            if lsblk_info is None:
+                # skip for now until better detection written
+                continue
+            current_fs = lsblk_info.get('FSTYPE')
+            if current_fs is None or len(current_fs) == 0:
+                # skip for now until better detection written
+                continue
+
+            # Blkid calls fat formats vfat and lsblk doesn't differentiate
+            # between fatsize, so if needed format is fat, only verify that the
+            # actual format is some kind of fat
+            if req_fs.startswith('fat'):
+                self.assertIn('fat', current_fs)
+            else:
+                self.assertEqual(req_fs, current_fs)
+
+    def _test_mount(self, conf, env):
+        """make sure fstab is okay and mounts are in right place"""
+        mount_configs = conf.find_matching('type', 'mount')
+        # do not try to verify that fstab exists if no mount points
+        if len(mount_configs) == 0:
+            return
+
+        # verify fstab exists and load data
+        fstab_path = os.path.join(env.tmpdir, 'fstab')
+        self.assertTrue(os.path.exists(env.tmpdir))
+        self.assertTrue(os.path.exists(fstab_path))
+        fstab = util.load_file(fstab_path).splitlines()
+
+        # check mountpoints
+        for conf_entry in mount_configs:
+            format_conf = conf.with_id(conf_entry.get('device'))
+            dev_path = conf.path_to(format_conf.get('volume'))
+            self.assertTrue(os.path.exists(dev_path))
+
+            mountpoint = conf_entry.get('path')
+            if format_conf.get('fstype') == 'swap':
+                mountpoint = 'none'
+
+            ident = None
+            if (conf.with_id(format_conf.get('volume')).get('type') in
+                    ['raid', 'bcache', 'disk', 'lvm_partition']):
+                ident = dev_path
+            else:
+                # anything else we mount by uuid
+                dev_uuid = block.get_volume_uuid(dev_path)
+                self.assertIsNotNone(dev_uuid)
+                ident = 'UUID=' + dev_uuid
+
+            try:
+                fstab_entry = next(i.split() for i in fstab if ident in i)
+            except StopIteration:
+                self.fail('fstab missing {}'.format(conf_entry['id']))
+            self.assertEqual(fstab_entry[1], mountpoint,
+                             'Fstabe entry for {} incorrect'
+                             .format(conf_entry['id']))
+
+
+class LvmTests(object):
+    """Verifier for lvm volume groups and lvm logical partitions"""
+    __test__ = False
+
+    def _test_lvm_vg(self, conf, env):
+        """ensure that all volgroups have required physical volumes"""
+        for volgroup in conf.find_matching('type', 'lvm_volgroup'):
+            req_paths = set(conf.path_to(dev) for dev in
+                            volgroup.get('devices'))
+            (out, _) = util.subp(['pvdisplay', '-C', '--separator', '=', '-o',
+                                  'vg_name,pv_name', '--noheadings'],
+                                 capture=True)
+            vg_name = volgroup.get('name')
+            self.assertIsNotNone(vg_name)
+            pv_paths = set(l.split('=')[-1] for l in out.strip().splitlines()
+                           if vg_name in l)
+            self.assertEqual(pv_paths, req_paths,
+                             'lvm volgroup {} requires devs: {}, only has {}'
+                             .format(vg_name, req_paths, pv_paths))
+
+    def _test_lvm_lv(self, conf, env):
+        """ensure that all needed logical volumes are present"""
+        for lvm_part in conf.find_matching('type', 'lvm_partition'):
+            lv_name = lvm_part.get('name')
+            vg_name = conf.with_id(lvm_part.get('volgroup')).get('name')
+            self.assertIsNotNone(lv_name)
+            self.assertIsNotNone(vg_name)
+            (out, _) = util.subp(['lvdisplay', '-C', '--separator', '=', '-o',
+                                  'lv_name,vg_name', '--noheadings'],
+                                 capture=True)
+            out = out.strip()
+
+            for line in out.splitlines():
+                line = line.strip()
+                (line_lv, line_vg) = line.split('=')
+                if line_lv == lv_name and line_vg == vg_name:
+                    break
+            else:
+                raise AssertionError("lvm lv '{}' missing in volgroup '{}'"
+                                     .format(lv_name, vg_name))
+
+
+class RaidTests(object):
+    """Verifier for mdadm configuration"""
+    __test__ = False
+
+    def _test_mdadm(self, conf, env):
+        """ensure all mdadm devices present and using correct devices"""
+        for conf_entry in conf.find_matching('type', 'raid'):
+            req_devs = list(conf.path_to(i) for i in conf_entry.get('devices'))
+            req_spare = list(conf.path_to(i) for i in
+                             conf_entry.get('spare_devices'))
+            raidlevel = conf_entry.get('raidlevel')
+            md_path = conf.path_to(conf_entry)
+            # FIXME: this is a bit of a hack, should later be switched to use
+            #        mdadm.md_block_until_sync()
+            error = None
+            for _ in range(20):
+                time.sleep(1)
+                try:
+                    block.mdadm.md_check(md_path, raidlevel, req_devs,
+                                         req_spare)
+                    break
+                except ValueError as _error:
+                    error = _error
+                    continue
+            else:
+                # md_check failed after 20 seconds, fail test
+                self.fail('md_check failed with exception {}'.format(error))
+
+
+class BcacheTests(object):
+    """Verifier for bcache configuration"""
+    __test__ = False
+
+    def _test_bcache(self, conf, env):
+        """ensure that correct cache and backing devices are used"""
+        for conf_entry in conf.find_matching('type', 'bcache'):
+            cache_kname = conf.kname_of(conf_entry['cache_device'])
+            backing_kname = conf.kname_of(conf_entry['backing_device'])
+            bcache_path = conf.path_to(conf_entry)
+            bcache_kname = conf.kname_of(conf_entry)
+            self.assertTrue(block.is_valid_device(bcache_path))
+            cache_slaves = os.listdir(
+                os.path.join('/sys/block', bcache_kname, 'slaves'))
+            self.assertEqual(set((cache_kname, backing_kname)),
+                             set(cache_slaves))
+
+            # since registering and then unregistering a bcache device too
+            # quickly can cause a kernel panic, sleep for 1 second to make it
+            # less likely that a kernel panic will occur when clear_holders
+            # runs on the next storage test
+            time.sleep(1)

=== modified file 'tests/unittests/test_reporter.py'
--- tests/unittests/test_reporter.py	2016-06-14 13:54:46 +0000
+++ tests/unittests/test_reporter.py	2016-09-15 18:06:47 +0000
@@ -147,8 +147,7 @@
         event_dict = self._get_reported_event(mock_report_event).as_dict()
         self.assertEqual(event_dict.get('name'), self.ev_name)
         self.assertEqual(event_dict.get('level'), 'INFO')
-        self.assertEqual(event_dict.get('description'),
-                         'started: ' + self.ev_desc)
+        self.assertEqual(event_dict.get('description'), self.ev_desc)
         self.assertEqual(event_dict.get('event_type'), events.START_EVENT_TYPE)
 
     @patch('curtin.reporter.events.report_event')
@@ -157,8 +156,7 @@
         event = self._get_reported_event(mock_report_event)
         self.assertIsInstance(event, events.FinishReportingEvent)
         event_dict = event.as_dict()
-        self.assertEqual(event_dict.get('description'),
-                         'finished: ' + self.ev_desc)
+        self.assertEqual(event_dict.get('description'), self.ev_desc)
 
     @patch('curtin.reporter.events.report_event')
     def test_report_finished_event_levelset(self, mock_report_event):
@@ -166,15 +164,13 @@
                                    result=events.status.FAIL)
         event_dict = self._get_reported_event(mock_report_event).as_dict()
         self.assertEqual(event_dict.get('level'), 'ERROR')
-        self.assertEqual(event_dict.get('description'),
-                         'failed: ' + self.ev_desc)
+        self.assertEqual(event_dict.get('description'), self.ev_desc)
 
         events.report_finish_event(self.ev_name, self.ev_desc,
                                    result=events.status.WARN)
         event_dict = self._get_reported_event(mock_report_event).as_dict()
         self.assertEqual(event_dict.get('level'), 'WARN')
-        self.assertEqual(event_dict.get('description'),
-                         'failed: ' + self.ev_desc)
+        self.assertEqual(event_dict.get('description'), self.ev_desc)
 
     @patch('curtin.reporter.events.report_event')
     def test_report_finished_post_files(self, mock_report_event):

=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py	2016-08-05 12:22:55 +0000
+++ tests/vmtests/__init__.py	2016-09-15 18:06:47 +0000
@@ -15,6 +15,7 @@
 import curtin.net as curtin_net
 import curtin.util as util
 
+from tools.report_webhook_logger import CaptureReporting
 from curtin.commands.install import INSTALL_PASS_MSG
 
 from .image_sync import query as imagesync_query
@@ -306,6 +307,26 @@
         self.success_file = os.path.join(self.logs, "success")
         self.errors_file = os.path.join(self.logs, "errors.json")
 
+        # write userdata
+        self.write_userdata(user_data)
+
+        # create target disk
+        logger.debug('Creating target disk')
+        self.target_disk = os.path.join(self.disks, "install_disk.img")
+        subprocess.check_call(["qemu-img", "create", "-f", TARGET_IMAGE_FORMAT,
+                              self.target_disk, "10G"],
+                              stdout=DEVNULL, stderr=subprocess.STDOUT)
+
+        # create output disk, mount ro
+        logger.debug('Creating output disk')
+        self.output_disk = os.path.join(self.boot, OUTPUT_DISK_NAME)
+        subprocess.check_call(["qemu-img", "create", "-f", TARGET_IMAGE_FORMAT,
+                              self.output_disk, "10M"],
+                              stdout=DEVNULL, stderr=subprocess.STDOUT)
+        subprocess.check_call(["mkfs.ext2", "-F", self.output_disk],
+                              stdout=DEVNULL, stderr=subprocess.STDOUT)
+
+    def write_userdata(self, user_data):
         # write cloud-init for installed system
         meta_data_file = os.path.join(self.install, "meta-data")
         with open(meta_data_file, "w") as fp:
@@ -314,13 +335,6 @@
         with open(user_data_file, "w") as fp:
             fp.write(user_data)
 
-        # create target disk
-        logger.debug('Creating target disk')
-        self.target_disk = os.path.join(self.disks, "install_disk.img")
-        subprocess.check_call(["qemu-img", "create", "-f", TARGET_IMAGE_FORMAT,
-                              self.target_disk, "10G"],
-                              stdout=DEVNULL, stderr=subprocess.STDOUT)
-
         # create seed.img for installed system's cloud init
         logger.debug('Creating seed disk')
         self.seed_disk = os.path.join(self.boot, "seed.img")
@@ -328,15 +342,6 @@
                               user_data_file, meta_data_file],
                               stdout=DEVNULL, stderr=subprocess.STDOUT)
 
-        # create output disk, mount ro
-        logger.debug('Creating output disk')
-        self.output_disk = os.path.join(self.boot, OUTPUT_DISK_NAME)
-        subprocess.check_call(["qemu-img", "create", "-f", TARGET_IMAGE_FORMAT,
-                              self.output_disk, "10M"],
-                              stdout=DEVNULL, stderr=subprocess.STDOUT)
-        subprocess.check_call(["mkfs.ext2", "-F", self.output_disk],
-                              stdout=DEVNULL, stderr=subprocess.STDOUT)
-
     def collect_output(self):
         logger.debug('extracting output disk')
         subprocess.check_call(['tar', '-C', self.collect, '-xf',
@@ -412,7 +417,8 @@
             dowait = "--dowait"
 
         # create launch cmd
-        cmd = ["tools/launch", "--arch=" + cls.arch, "-v", dowait]
+        cmd = ["tools/launch", "--arch=" + cls.arch, "-v", dowait,
+               "--mem=1024"]
         if not cls.interactive:
             cmd.extend(["--silent", "--power=off"])
 
@@ -516,6 +522,30 @@
                 fp.write(json.dumps({'grub': {'update_nvram': True}}))
             configs.append(grub_config)
 
+        # set reporting logger
+        cls.reporting_log = os.path.join(cls.td.logs, 'webhooks-events.json')
+        reporting_logger = CaptureReporting(cls.reporting_log)
+
+        # write reporting config
+        reporting_config = os.path.join(cls.td.install, 'reporting.cfg')
+        localhost_url = ('http://' + get_lan_ip() +
+                         ':{:d}/'.format(reporting_logger.port))
+        with open(reporting_config, 'w') as fp:
+            fp.write(json.dumps({
+                'install': {
+                    'log_file': '/tmp/install.log',
+                    'post_files': ['/tmp/install.log'],
+                },
+                'reporting': {
+                    'maas': {
+                        'level': 'DEBUG',
+                        'type': 'webhook',
+                        'endpoint': localhost_url,
+                    },
+                },
+            }))
+        configs.append(reporting_config)
+
         if cls.multipath:
             disks = disks * cls.multipath_num_paths
 
@@ -530,9 +560,10 @@
         logger.info('Running curtin installer: {}'.format(cls.install_log))
         try:
             with open(lout_path, "wb") as fpout:
-                cls.boot_system(cmd, timeout=cls.install_timeout,
-                                console_log=cls.install_log, proc_out=fpout,
-                                purpose="install")
+                with reporting_logger:
+                    cls.boot_system(cmd, timeout=cls.install_timeout,
+                                    console_log=cls.install_log,
+                                    proc_out=fpout, purpose="install")
         except TimeoutExpired:
             logger.error('Curtin installer failed with timeout')
             cls.tearDownClass()
@@ -800,6 +831,25 @@
                     self.assertIn(link, contents)
                 self.assertIn(diskname, contents)
 
+    def test_reporting_data(self):
+        with open(self.reporting_log, 'r') as fp:
+            data = json.load(fp)
+        self.assertTrue(len(data) > 0)
+        first_event = data[0]
+        self.assertEqual(first_event['event_type'], 'start')
+        next_event = data[1]
+        # make sure we don't have that timestamp bug
+        self.assertNotEqual(first_event['timestamp'], next_event['timestamp'])
+        final_event = data[-1]
+        self.assertEqual(final_event['event_type'], 'finish')
+        self.assertEqual(final_event['name'], 'cmd-install')
+        # check for install log
+        [events_with_files] = [ev for ev in data if 'files' in ev]
+        self.assertIn('files',  events_with_files)
+        [files] = events_with_files.get('files', [])
+        self.assertIn('path', files)
+        self.assertEqual('/tmp/install.log', files.get('path', ''))
+
     def test_interfacesd_eth0_removed(self):
         """ Check that curtin has removed /etc/network/interfaces.d/eth0.cfg
             by examining the output of a find /etc/network > find_interfaces.d
@@ -914,6 +964,9 @@
     def test_interfacesd_eth0_removed(self):
         pass
 
+    def test_reporting_data(self):
+        pass
+
     def _maybe_raise(self, exc):
         if self.allow_test_fails:
             raise exc
@@ -1011,6 +1064,9 @@
     collect_post = textwrap.dedent(
         'tar -C "%s" -cf "%s" .' % (output_dir, output_device))
 
+    # copy /root for curtin config and install.log
+    copy_rootdir = textwrap.dedent("cp -a /root " + output_dir)
+
     # failsafe poweroff runs on precise only, where power_state does
     # not exist.
     precise_poweroff = textwrap.dedent("""#!/bin/sh -x
@@ -1018,8 +1074,8 @@
         shutdown -P now "Shutting down on precise"
         """)
 
-    scripts = ([collect_prep] + collect_scripts + [collect_post] +
-               [precise_poweroff])
+    scripts = ([collect_prep] + [copy_rootdir] + collect_scripts +
+               [collect_post] + [precise_poweroff])
 
     for part in scripts:
         if not part.startswith("#!"):
@@ -1093,5 +1149,14 @@
     return ret
 
 
+def get_lan_ip():
+    out = subprocess.check_output(['ip', 'addr'])
+    out = out.decode()
+    line = next(l for l in out.splitlines() if 'inet' in l and 'global' in l)
+    addr = line.split()[1]
+    if '/' in addr:
+        addr = addr[:addr.index('/')]
+    return addr
+
 apply_keep_settings()
 logger = _initialize_logging()

=== modified file 'tests/vmtests/image_sync.py'
--- tests/vmtests/image_sync.py	2016-06-20 17:10:03 +0000
+++ tests/vmtests/image_sync.py	2016-09-15 18:06:47 +0000
@@ -400,7 +400,7 @@
                     verbosity=vlevel)
     try:
         if args.output_format == FORMAT_JSON:
-            print(util.json_dumps(results).decode())
+            print(util.json_dumps(results))
         else:
             output = []
             for item in results:

=== added file 'tools/__init__.py'
=== added file 'tools/curtin-log-print'
--- tools/curtin-log-print	1970-01-01 00:00:00 +0000
+++ tools/curtin-log-print	2016-09-15 18:06:47 +0000
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+#   Copyright (C) 2016 Canonical Ltd.
+#
+#   Author: Ryan Harper <[e-mail]>
+#
+#   Curtin is free software: you can redistribute it and/or modify it under
+#   the terms of the GNU Affero General Public License as published by the
+#   Free Software Foundation, either version 3 of the License, or (at your
+#   option) any later version.
+#
+#   Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
+#   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+#   FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for
+#   more details.
+#
+#   You should have received a copy of the GNU Affero General Public License
+#   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
+import argparse
+import datetime
+import json
+import sys
+import base64
+
+
+#  An event:
+'''
+{
+        "description": "executing late commands",
+        "event_type": "start",
+        "level": "INFO",
+        "name": "cmd-install/stage-late"
+        "origin": "curtin",
+        "timestamp": 1461164249.1590767,
+},
+
+    {
+        "description": "executing late commands",
+        "event_type": "finish",
+        "level": "INFO",
+        "name": "cmd-install/stage-late",
+        "origin": "curtin",
+        "result": "SUCCESS",
+        "timestamp": 1461164249.1590767
+    }
+
+'''
+format_key = {
+    '%d': 'delta',
+    '%D': 'description',
+    '%e': 'event_type',
+    '%l': 'level',
+    '%n': 'name',
+    '%o': 'origin',
+    '%r': 'result',
+    '%t': 'timestamp',
+}
+
+formatting_help = " ".join(["{}: {}".format(k.replace('%', '%%'), v)
+                           for k, v in format_key.items()])
+
+
+def format_record(msg, event):
+    for i, j in format_key.items():
+        if i in msg:
+            msg = msg.replace(i, "{%s}" % j)
+    return msg.format(**event)
+
+
+def dump_event_files(event):
+    content = {k: v for k, v in event.items() if k not in ['content']}
+    files = content['files']
+    for f in files:
+        fname = f['path']
+        fcontent = base64.b64decode(f['content']).decode('ascii')
+        print("%s:\n%s" % (fname, fcontent))
+
+
+def generate_records(j, blame_sort=False, print_format="%d seconds in %D",
+                     dump_files=False):
+    records = []
+    timestamps = {}
+    total_time = 0
+    for event in j:
+        name = event.get('name')
+        if 'files' in event:
+            print('Event with files: %s %s @ %s' % (event['event_type'],
+                                                    event['name'],
+                                                    event['timestamp']))
+            if dump_files:
+                dump_event_files(event)
+
+        if event['event_type'] == 'start':
+            timestamps[name] = {'start': event['timestamp']}
+        else:
+            timestamps[name].update({'finish': event['timestamp']})
+            start = datetime.datetime.utcfromtimestamp(
+                timestamps[name]['start'])
+            end = datetime.datetime.utcfromtimestamp(
+                timestamps[name]['finish'])
+            delta = end - start
+            total_time += delta.total_seconds()
+            event['delta'] = "{:08.5f}".format(delta.total_seconds())
+            records.append(format_record(print_format, event))
+
+    records.append(' ---\n%3.5f seconds total time' % total_time)
+    return records
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description='curtin-print-log - pretty print and sort curtin logs',
+        prog='curtin-print-log')
+    parser.add_argument('--blame', action='store_true',
+                        default=False,
+                        dest='blame_sort',
+                        help='sort events by total time.')
+    parser.add_argument('--dumpfiles', action='store_true',
+                        default=False,
+                        dest='dump_files',
+                        help='dump content of any posted files')
+    parser.add_argument('--format', action='store',
+                        dest='print_format',
+                        default='%d seconds in %D',
+                        help='specify formatting of output. ' +
+                        formatting_help)
+    parser.add_argument('infile', nargs='?', type=argparse.FileType('r'),
+                        help='Path to log to parse. Use - for stdin')
+
+    opts = parser.parse_args(sys.argv[1:])
+    if not opts.infile:
+        parser.print_help()
+        sys.exit(1)
+
+    try:
+        j = json.load(opts.infile)
+    except json.JSONDecodeError:
+        print("Input must be valid JSON")
+        sys.exit(1)
+
+    records = generate_records(j, blame_sort=opts.blame_sort,
+                               print_format=opts.print_format,
+                               dump_files=opts.dump_files)
+    summary = []
+    if opts.blame_sort is True:
+        summary = records[-1:]
+        records = sorted(records[:-1], reverse=True)
+
+    print("\n".join(records + summary))
+
+
+if __name__ == '__main__':
+    main()

=== modified file 'tools/launch'
--- tools/launch	2016-07-15 16:14:53 +0000
+++ tools/launch	2016-09-15 18:06:47 +0000
@@ -37,7 +37,7 @@
       -h | --help       show this message
       -i | --initrd F   use initramfs F
       -k | --kernel F   use kernel K
-           --mem    K   memory in Kb
+           --mem    M   memory in Mb
       -n | --netdev     netdev can be 'user' or a bridge
       -p | --publish F  make file 'F' available in web server
            --silent     use -nographic

=== removed file 'tools/report-webhook-logger'
--- tools/report-webhook-logger	2015-10-09 14:47:09 +0000
+++ tools/report-webhook-logger	1970-01-01 00:00:00 +0000
@@ -1,100 +0,0 @@
-#!/usr/bin/python3
-try:
-    # python2
-    import SimpleHTTPServer as http_server
-    import SocketServer as socketserver
-except ImportError:
-    import http.server as http_server
-    import socketserver
-
-import json
-import sys
-
-EXAMPLE_CONFIG = """\
-# example config
-reporting:
-  mypost:
-    type: webhook
-    endpoint: %(endpoint)s
-install:
-  log_file: /tmp/foo
-  post_files: [/tmp/foo]
-
-# example python:
-from curtin.reporter import events, update_configuration
-cfg = {'mypost': {'type': 'webhook', 'endpoint': '%(endpoint)s'}}
-update_configuration(cfg)
-with events.ReportEventStack(name="myname", description="mydesc",
-                             reporting_enabled=True):
-    print("do something")
-"""
-
-if len(sys.argv) > 2:
-    PORT = int(sys.argv[2])
-    addr = sys.argv[1]
-elif len(sys.argv) > 1:
-    PORT = int(sys.argv[1])
-    addr = ""
-else:
-    PORT = 8000
-    addr = ""
-
-
-def render_event_string(event_str):
-    return json.dumps(json.loads(event_str), indent=1)
-
-
-class ServerHandler(http_server.SimpleHTTPRequestHandler):
-
-    def log_request(self, code, size=None):
-        lines = [
-            "== %s %s ==" % (self.command, self.path),
-            str(self.headers).replace('\r', '')]
-        if self._message:
-            lines.append(self._message)
-        sys.stdout.write('\n'.join(lines) + '\n')
-        sys.stdout.flush()
-
-    def do_GET(self):
-        self._message = None
-        self.send_response(200)
-        self.end_headers()
-        self.wfile.write("content of %s\n" % self.path)
-
-    def do_POST(self):
-        length = int(self.headers['Content-Length'])
-        post_data = self.rfile.read(length).decode('utf-8')
-        try:
-            self._message = render_event_string(post_data)
-        except Exception as e:
-            self._message = '\n'.join(
-		["failed printing event: %s" % e, post_data])
-
-        msg = "received post to %s" % self.path
-        self.send_response(200)
-        self.send_header("Content-type", "text/plain")
-        self.end_headers()
-        self.wfile.write(msg.encode('utf-8'))
-
-# avoid 'Address already in use' after ctrl-c
-socketserver.TCPServer.allow_reuse_address = True
-
-Handler = ServerHandler
-httpd = socketserver.TCPServer(("", PORT), Handler)
-httpd.allow_reuse_address = True
-
-info = {
-    'interface': addr or "localhost",
-    'port': PORT,
-    'endpoint': "http://" + (addr or "localhost") + ":%s" % PORT
-}
-print("Serving at: %(endpoint)s" % info)
-print("Post to this with:\n%s\n" % (EXAMPLE_CONFIG % info))
-
-try:
-    httpd.serve_forever()
-except KeyboardInterrupt:
-    sys.stdout.flush()
-    pass
-httpd.server_close()
-sys.exit(0)

=== added file 'tools/report_webhook_logger.py'
--- tools/report_webhook_logger.py	1970-01-01 00:00:00 +0000
+++ tools/report_webhook_logger.py	2016-09-15 18:06:47 +0000
@@ -0,0 +1,174 @@
+#!/usr/bin/python3
+try:
+    # python2
+    import SimpleHTTPServer as http_server
+    import SocketServer as socketserver
+except ImportError:
+    import http.server as http_server
+    import socketserver
+
+import json
+import os
+import sys
+import threading
+
+EXAMPLE_CONFIG = """\
+# example config
+reporting:
+  mypost:
+    type: webhook
+    endpoint: %(endpoint)s
+install:
+  log_file: /tmp/foo
+  post_files: [/tmp/foo]
+
+# example python:
+from curtin.reporter import events, update_configuration
+cfg = {'mypost': {'type': 'webhook', 'endpoint': '%(endpoint)s'}}
+update_configuration(cfg)
+with events.ReportEventStack(name="myname", description="mydesc",
+                             reporting_enabled=True):
+    print("do something")
+"""
+
+CURTIN_EVENTS = []
+DEFAULT_PORT = 8000
+addr = ""
+
+
+def render_event_string(event_str):
+    return json.dumps(json.loads(event_str), indent=1)
+
+
+def write_event_string(target, event_str):
+    try:
+        with open(target, 'r') as fp:
+            data = json.load(fp)
+    except:
+        data = []
+    data.append(json.loads(event_str))
+    with open(target, 'w') as fp:
+        json.dump(data, fp)
+
+
+class ServerHandler(http_server.SimpleHTTPRequestHandler):
+    result_log_file = None
+
+    def log_request(self, code, size=None):
+        if self.result_log_file:
+            return
+        lines = [
+            "== %s %s ==" % (self.command, self.path),
+            str(self.headers).replace('\r', '')]
+        if self._message:
+            lines.append(self._message)
+        sys.stdout.write('\n'.join(lines) + '\n')
+        sys.stdout.flush()
+
+    def do_GET(self):
+        self._message = None
+        self.send_response(200)
+        self.end_headers()
+        self.wfile.write("content of %s\n" % self.path)
+
+    def do_POST(self):
+        length = int(self.headers['Content-Length'])
+        post_data = self.rfile.read(length).decode('utf-8')
+        try:
+            if self.result_log_file:
+                write_event_string(self.result_log_file, post_data)
+            self._message = render_event_string(post_data)
+        except Exception as e:
+            self._message = '\n'.join(
+                ["failed printing event: %s" % e, post_data])
+
+        msg = "received post to %s" % self.path
+        self.send_response(200)
+        self.send_header("Content-type", "text/plain")
+        self.end_headers()
+        self.wfile.write(msg.encode('utf-8'))
+
+
+def GenServerHandlerWithResultFile(file_path):
+    class ExtendedServerHandler(ServerHandler):
+        result_log_file = file_path
+    return ExtendedServerHandler
+
+
+def get_httpd(port=None, result_file=None):
+    # avoid 'Address already in use' after ctrl-c
+    socketserver.TCPServer.allow_reuse_address = True
+
+    # get first available port if none specified
+    if port is None:
+        port = 0
+
+    if result_file:
+        Handler = GenServerHandlerWithResultFile(result_file)
+    else:
+        Handler = ServerHandler
+    httpd = socketserver.TCPServer(("", port), Handler)
+    httpd.allow_reuse_address = True
+
+    return httpd
+
+
+def run_server(port=DEFAULT_PORT, log_data=True):
+    """Run the server and capture output, redirecting output to /dev/null if
+       log_data = False"""
+    httpd = get_httpd(port=port)
+
+    _stdout = sys.stdout
+    with open(os.devnull, 'w') as fp:
+        try:
+            if not log_data:
+                sys.stdout = fp
+            httpd.serve_forever()
+        except KeyboardInterrupt:
+            sys.stdout.flush()
+            pass
+        finally:
+            sys.stdout = _stdout
+            httpd.server_close()
+
+    return CURTIN_EVENTS
+
+
+class CaptureReporting:
+
+    def __init__(self, result_file):
+        self.result_file = result_file
+        self.httpd = get_httpd(result_file=self.result_file,
+                               port=None)
+        self.httpd.server_activate()
+        (self.bind_addr, self.port) = self.httpd.server_address
+
+    def __enter__(self):
+        if os.path.exists(self.result_file):
+            os.remove(self.result_file)
+        self.worker = threading.Thread(target=self.httpd.serve_forever)
+        self.worker.start()
+        return self
+
+    def __exit__(self, etype, value, trace):
+        self.httpd.shutdown()
+
+
+if __name__ == "__main__":
+    if len(sys.argv) > 2:
+        port = int(sys.argv[2])
+        addr = sys.argv[1]
+    elif len(sys.argv) > 1:
+        port = int(sys.argv[1])
+        addr = ""
+    else:
+        port = DEFAULT_PORT
+    info = {
+        'interface': addr or "localhost",
+        'port': port,
+        'endpoint': "http://" + (addr or "localhost") + ":%s" % port
+    }
+    print("Serving at: %(endpoint)s" % info)
+    print("Post to this with:\n%s\n" % (EXAMPLE_CONFIG % info))
+    run_server(port=port, log_data=True)
+    sys.exit(0)

=== modified file 'tools/run-pep8'
--- tools/run-pep8	2015-03-09 16:32:36 +0000
+++ tools/run-pep8	2016-09-15 18:06:47 +0000
@@ -1,6 +1,11 @@
 #!/bin/bash
 
-pycheck_dirs=( "curtin/" "tests/" )
+pycheck_dirs=(
+    "curtin/"
+    "tests/"
+    "tools/curtin-log-print"
+    "tools/report_webhook_logger.py"
+)
 bin_files=( )
 CR="
 "

=== modified file 'tools/run-pyflakes'
--- tools/run-pyflakes	2016-02-12 16:43:18 +0000
+++ tools/run-pyflakes	2016-09-15 18:06:47 +0000
@@ -4,10 +4,20 @@
 CR="
 "
 vmtests=""
+storagetest_runner=""
 if [ "$PYTHON_VERSION" = "3" ]; then
     vmtests="tests/vmtests/"
+    storagetest_runner="tests/storagetest_runner/"
 fi
-pycheck_dirs=( "curtin/" "tests/unittests/" $vmtests )
+pycheck_dirs=(
+    "curtin/"
+    "tests/unittests/"
+    $vmtests
+    $storagetest_runner
+    "tests/storagetests/"
+    "tools/curtin-log-print"
+    "tools/report_webhook_logger.py"
+)
 
 set -f
 if [ $# -eq 0 ]; then