Merge ~raharper/curtin:fix/vmtest-dont-clobber-cls-collect-scripts into curtin:master

[Ryan Harper]

vmtests: prevent tests from modifying cls.collect_scripts

vmtests now has a set of collect scripts which are shared in all
subclassed tests and should not be modifed by the subclass. Fix
test_lvm_raid.py which modified cls.collect_scripts to use
extra_collect_scripts and update the base class to raise a RuntimeError
if a subclass presents a modified cls.collect_scripts.

[Ryan Harper (raharper)]

vmtest run started here:

https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-debug/104/console

This will fix up current curtin-vmtest failures.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:6a6f14202faede2a86ef5e7dba4138814d13e2eb
https://jenkins.ubuntu.com/server/job/curtin-ci/1065/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/1065
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/1065
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/1065/console
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/1065

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/1065/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

2 errors and 1 failure:

Ran 3333 tests in 21326.658s

FAILED (SKIP=384, errors=2, failures=1)
Mon, 24 Sep 2018 21:22:32 +0000: vmtest end [1] in 21335s

The two failures were the MultipathBasic tests which are known to fail from time to time.

The failure was a missing DNS entry in test_network_vlan.TrustyHWEXTestNetworkVlan, which we've seen before.

I'll repeat this vmtest on those 3 tests and see if we get positive results.

[Ryan Harper (raharper)]

And we have success on repeat:

Ran 33 tests in 291.964s

OK (SKIP=6)
Mon, 24 Sep 2018 22:00:19 +0000: vmtest end [0] in 299s

https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-debug/105/console

[Ryan Harper (raharper)]

I'm going to mark approved to land this so we don't break jenkins storage again tonight.

review: Approve

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

[Scott Moser (smoser)]

we do need some better way of doing this... collect needs just to be more general purpose.
but for now this is good enough.

thank you.

review: Approve

[Server Team CI bot (server-team-bot)]

FAILED: Autolanding.
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/curtin-autoland-test/28/
Executed test runs:
    None: https://jenkins.ubuntu.com/server/job/admin-lp-git-autoland/107/console

review: Needs Fixing (continuous-integration)

diff --git a/tests/vmtests/__init__.py b/tests/vmtests/__init__.py
index 7e31491..af75599 100644
--- a/tests/vmtests/__init__.py
+++ b/tests/vmtests/__init__.py
@@ -831,6 +831,9 @@ class VMBaseClass(TestCase):
             cls.collect_scripts = (
                 DEFAULT_COLLECT_SCRIPTS['common'] +
                 DEFAULT_COLLECT_SCRIPTS[cls.target_distro])
+        else:
+            raise RuntimeError('cls collect scripts not empty: %s' %
+                               cls.collect_scripts)
 
         # append extra from subclass
         if cls.extra_collect_scripts:
diff --git a/tests/vmtests/test_lvm_raid.py b/tests/vmtests/test_lvm_raid.py
index 0c50941..99a33f2 100644
--- a/tests/vmtests/test_lvm_raid.py
+++ b/tests/vmtests/test_lvm_raid.py
@@ -14,8 +14,9 @@ class TestLvmOverRaidAbs(TestMdadmAbs, TestLvmAbs):
     dirty_disks = True
     extra_disks = ['10G'] * 4
 
-    collect_scripts = TestLvmAbs.collect_scripts
-    collect_scripts += TestMdadmAbs.collect_scripts + [textwrap.dedent("""
+    extra_collect_scripts = TestLvmAbs.extra_collect_scripts
+    extra_collect_scripts += TestMdadmAbs.extra_collect_scripts
+    extra_collect_scripts += [textwrap.dedent("""
         cd OUTPUT_COLLECT_D
         ls -al /dev/md* > dev_md
         cp -a /etc/mdadm etc_mdadm