Merge lp:~raharper/curtin/trunk.vmtest-tar-sparse-disks into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

vmtests: add option to tar disk images after test run

vmtests may create a number of disk image files for use, these
are all sparse disks which report a size larger than the actual
space they occupy. In some cases someone copying files may
not be aware of the sparseness and expand the images instead.
Setting CURTIN_VMTEST_TAR_DISKS=1 will trigger vmtest to tar
sparsely the files and write out a .tar file in the same path.

This flag is disabled by default.

[lp:~raharper/curtin/trunk.vmtest-tar-sparse-disks]

528. By Ryan Harper on 2017-09-28

Add docs for CURTIN_VMTEST_TAR_DISKS

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:528
https://jenkins.ubuntu.com/server/job/curtin-ci/632/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/632
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/632
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/632
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/632

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/632/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

nits. but approve with those changes.

review: Approve

[lp:~raharper/curtin/trunk.vmtest-tar-sparse-disks]

529. By Ryan Harper on 2017-09-28

Reduce vertical space used

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:529
https://jenkins.ubuntu.com/server/job/curtin-ci/633/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/633
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/633
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/633
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/633

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/633/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.vmtest-tar-sparse-disks]

530. By Ryan Harper on 2017-09-28

Drop quotes from CURTIN_VMTEST_TAR_DISKS variable

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:530
https://jenkins.ubuntu.com/server/job/curtin-ci/634/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/634
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/634
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/634
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/634

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/634/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.vmtest-tar-sparse-disks]

531. By Ryan Harper on 2017-09-28

Change language around values that trigger behavior

532. By Ryan Harper on 2017-09-28

Grammar fix

=== modified file 'doc/topics/integration-testing.rst'
--- doc/topics/integration-testing.rst	2017-05-19 20:26:31 +0000
+++ doc/topics/integration-testing.rst	2017-09-28 21:15:09 +0000
@@ -161,6 +161,12 @@
     - ``logs``: install and boot logs
     - ``collect``: data collected by the boot phase
 
+- ``CURTIN_VMTEST_TAR_DISKS``: default 0
+
+  Vmtest writes out disk image files sparsely into a disks directory
+  If this flag is set to a non-zero number, vmtest will tar all disks in
+  the directory into a single disks.tar and remove the sparse disk files.
+
 - ``CURTIN_VMTEST_TOPDIR``: default $TMPDIR/vmtest-<timestamp>
 
   Vmtest puts all test data under this value.  By default, it creates

=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py	2017-09-01 21:57:13 +0000
+++ tests/vmtests/__init__.py	2017-09-28 21:15:09 +0000
@@ -38,6 +38,7 @@
 CURTIN_VMTEST_IMAGE_SYNC = os.environ.get("CURTIN_VMTEST_IMAGE_SYNC", "1")
 IMAGE_SYNCS = []
 TARGET_IMAGE_FORMAT = "raw"
+TAR_DISKS = bool(int(os.environ.get("CURTIN_VMTEST_TAR_DISKS", "0")))
 
 
 DEFAULT_BRIDGE = os.environ.get("CURTIN_VMTEST_BRIDGE", "user")
@@ -938,7 +939,8 @@
         clean_working_dir(cls.td.tmpdir, success,
                           keep_pass=KEEP_DATA['pass'],
                           keep_fail=KEEP_DATA['fail'])
-
+        if TAR_DISKS:
+            tar_disks(cls.td.tmpdir)
         cls.cleanIscsiState(success,
                             keep_pass=KEEP_DATA['pass'],
                             keep_fail=KEEP_DATA['fail'])
@@ -1424,6 +1426,23 @@
     KEEP_DATA.update(data)
 
 
+def tar_disks(tmpdir, outfile="disks.tar", diskmatch=".img"):
+    """ Tar up files in ``tmpdir``/disks that ends with the pattern supplied"""
+
+    disks_dir = os.path.join(tmpdir, "disks")
+    if os.path.exists(disks_dir):
+        outfile = os.path.join(disks_dir, outfile)
+        disks = [os.path.join(disks_dir, disk) for disk in
+                 os.listdir(disks_dir) if disk.endswith(diskmatch)]
+        cmd = ["tar", "--create", "--file=%s" % outfile,
+               "--verbose", "--remove-files", "--sparse"]
+        cmd.extend(disks)
+        logger.info('Taring %s disks sparsely to %s', len(disks), outfile)
+        util.subp(cmd, capture=True)
+    else:
+        logger.error('Failed to find "disks" dir under tmpdir: %s', tmpdir)
+
+
 def boot_log_wrap(name, func, cmd, console_log, timeout, purpose):
     logger.debug("%s[%s]: booting with timeout=%s log=%s cmd: %s",
                  name, purpose, timeout, console_log, ' '.join(cmd))

=== modified file 'tools/jenkins-runner'
--- tools/jenkins-runner	2017-08-03 15:36:52 +0000
+++ tools/jenkins-runner	2017-09-28 21:15:09 +0000
@@ -4,6 +4,7 @@
 pkeep=${CURTIN_VMTEST_KEEP_DATA_PASS:-logs,collect}
 fkeep=${CURTIN_VMTEST_KEEP_DATA_FAIL:-logs,collect}
 reuse=${CURTIN_VMTEST_REUSE_TOPDIR:-0}
+export CURTIN_VMTEST_TAR_DISKS=${CURTIN_VMTEST_TAR_DISKS:-0}
 export CURTIN_VMTEST_REUSE_TOPDIR=$reuse
 export CURTIN_VMTEST_IMAGE_SYNC=${CURTIN_VMTEST_IMAGE_SYNC:-0}
 export CURTIN_VMTEST_KEEP_DATA_PASS=$pkeep