Merge ~smoser/curtin:feature/add-fstest-battery into curtin:master

[Scott Moser]

vmtest: Add Filesystem Battery test.

This test exercises filesystem creation, mount and unmount of all
supported filesystem types.

Also here is a fix for jfs filesystem creation. mkfs_jfs requires a '-q'
argument to force/quiet. Otherwise it will prompt to ask if you
really want to do that.

see commit message

[Scott Moser (smoser)]

I still need to test that the things are mountable. So far I've only been through the installation of Bionic.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:a6d0ba0f13de5de229eee7e55f4541cc7539b9e2
https://jenkins.ubuntu.com/server/job/curtin-ci/820/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/820
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/820
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/820
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/820/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/820/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:a6d0ba0f13de5de229eee7e55f4541cc7539b9e2
https://jenkins.ubuntu.com/server/job/curtin-ci/823/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/823
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/823
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/823
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/823

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/823/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:22bef4f07c387fe97aec3d85ed947f529a773a6a
https://jenkins.ubuntu.com/server/job/curtin-ci/835/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/835
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/835
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/835
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/835

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/835/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

And it all passes? Nice work. One comment on where to store the blkid output parser.

[Scott Moser (smoser)]

its not completely generic blkid parser as it is.
but yes, i thought about putting that into the base class and just always collecting that info.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:d4f0600d6379a86eb3cf866ed8f1c481aac7523d
https://jenkins.ubuntu.com/server/job/curtin-ci/864/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/864
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/864
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/864
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/864

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/864/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Let's confirm this passed on Jenkins and I'm +1

review: Approve

[Scott Moser (smoser)]

On diglet, I did:
 git

$ git rev-parse HEAD
ece93d5d40c56f5e07c3575f134de0e9ae915330

# full output: http://paste.ubuntu.com/p/GFZpHtjZ5k/
$ ./tools/jenkins-runner -p4 tests/vmtests/test_fs_battery.py
CURTIN_VMTEST_IMAGE_SYNC=0
CURTIN_VMTEST_ISCSI_PORTAL=10.245.168.20:5336
CURTIN_VMTEST_KEEP_DATA_FAIL=logs,collect
CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect
CURTIN_VMTEST_LOG=/srv/smoser/curtin/curtin/output/debug.log
CURTIN_VMTEST_PARALLEL=4
CURTIN_VMTEST_REUSE_TOPDIR=0
CURTIN_VMTEST_TAR_DISKS=0
CURTIN_VMTEST_TOPDIR=/srv/smoser/curtin/curtin/output
TGT_IPC_SOCKET=/srv/smoser/curtin/curtin/output/tgt.d/socket
TGT_LOG_D=/srv/smoser/curtin/curtin/output/tgt.d
TGT_PID=17545
TGT_PORTAL=10.245.168.20:5336
http_proxy=
https_proxy=
no_proxy=
Quering synced ephemeral images/kernels in /srv/images
======================================================================================
 Release Codename ImageDate Arch/SubArch Path
--------------------------------------------------------------------------------------
   12.04 precise 20170424 amd64/hwe-t precise/amd64/20170424/root-image.gz
   14.04 trusty 20180302 amd64/hwe-t trusty/amd64/20180302/squashfs
   14.04 trusty 20180302 amd64/hwe-x trusty/amd64/20180302/squashfs
   16.04 xenial 20180306 amd64/ga-16.04 xenial/amd64/20180306/squashfs
   16.04 xenial 20180306 amd64/hwe-16.04 xenial/amd64/20180306/squashfs
   16.04 xenial 20180306 amd64/hwe-16.04-edge xenial/amd64/20180306/squashfs
   17.04 zesty 20171219 amd64/ga-17.04 zesty/amd64/20171219/squashfs
   17.10 artful 20180303 amd64/ga-17.10 artful/amd64/20180303/squashfs
   18.04 bionic 20180224 amd64/ga-18.04 bionic/amd64/20180224/squashfs
======================================================================================

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:ece93d5d40c56f5e07c3575f134de0e9ae915330
https://jenkins.ubuntu.com/server/job/curtin-ci/868/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/868
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/868
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/868
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/868

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/868/rebuild

review: Approve (continuous-integration)

diff --git a/curtin/block/__init__.py b/curtin/block/__init__.py
index a82ed57..50e953e 100644
--- a/curtin/block/__init__.py
+++ b/curtin/block/__init__.py
@@ -1051,13 +1051,16 @@ def detect_required_packages_mapping():
                 'ext2': ['e2fsprogs'],
                 'ext3': ['e2fsprogs'],
                 'ext4': ['e2fsprogs'],
+                'jfs': ['jfsutils'],
                 'lvm_partition': ['lvm2'],
                 'lvm_volgroup': ['lvm2'],
+                'ntfs': ['ntfs-3g'],
                 'raid': ['mdadm'],
+                'reiserfs': ['reiserfsprogs'],
                 'xfs': ['xfsprogs'],
+                'zfsroot': ['zfsutils-linux', 'zfs-initramfs'],
                 'zfs': ['zfsutils-linux', 'zfs-initramfs'],
                 'zpool': ['zfsutils-linux', 'zfs-initramfs'],
-                'zfsroot': ['zfsutils-linux', 'zfs-initramfs'],
             },
         },
     }
diff --git a/curtin/block/mkfs.py b/curtin/block/mkfs.py
index 5cdb87c..a199d05 100644
--- a/curtin/block/mkfs.py
+++ b/curtin/block/mkfs.py
@@ -54,6 +54,7 @@ family_flag_mappings = {
     "force": {"btrfs": "--force",
               "ext": "-F",
               "fat": "-I",
+              "jfs": "-q",
               "ntfs": "--force",
               "reiserfs": "-f",
               "swap": "--force",
diff --git a/examples/tests/filesystem_battery.yaml b/examples/tests/filesystem_battery.yaml
new file mode 100644
index 0000000..ba4fcac
--- /dev/null
+++ b/examples/tests/filesystem_battery.yaml
@@ -0,0 +1,101 @@
+showtrace: true
+early_commands:
+  "00": [apt-get, update, -qy]
+  "01": [apt-get, install, -qy, --no-install-recommends,
+         ntfs-3g, jfsutils, reiserfsprogs]
+storage:
+  version: 1
+  config:
+    - id: disk1
+      type: disk
+      ptable: msdos
+      model: QEMU HARDDISK
+      serial: disk-a
+      wipe: superblock
+      grub_device: true
+    - id: disk1p1
+      type: partition
+      number: 1
+      size: 3GB
+      device: disk1
+      flag: boot
+    - id: disk1p1_fs
+      type: format
+      fstype: ext4
+      volume: disk1p1
+      label: 'cloudimg-rootfs'
+    - id: disk1p1_mount
+      type: mount
+      path: /
+      device: disk1p1_fs
+    - id: disk2
+      type: disk
+      serial: fsbattery
+      wipe: superblock
+      ptable: gpt
+    - {id: d2p01, number:  1, device: disk2, type: partition, size: 500M}
+    - {id: d2p02, number:  2, device: disk2, type: partition, size: 500M}
+    - {id: d2p03, number:  3, device: disk2, type: partition, size: 500M}
+    - {id: d2p04, number:  4, device: disk2, type: partition, size: 500M}
+    - {id: d2p05, number:  5, device: disk2, type: partition, size: 500M}
+    - {id: d2p06, number:  6, device: disk2, type: partition, size: 500M}
+    - {id: d2p07, number:  7, device: disk2, type: partition, size: 500M}
+    - {id: d2p08, number:  8, device: disk2, type: partition, size: 500M}
+    - {id: d2p09, number:  9, device: disk2, type: partition, size: 500M}
+    - {id: d2p10, number: 10, device: disk2, type: partition, size: 500M}
+    - id: fs01
+      type: format
+      fstype: btrfs
+      label: mybtrfs
+      volume: d2p01
+      uuid: 8946d6ad-1e5f-4609-924c-4a39b6b561c9
+    - id: fs02
+      type: format
+      fstype: ext2
+      label: myext2
+      volume: d2p02
+      uuid: 5d60e5e8-0c41-11e8-a664-525400123456
+    - id: fs03
+      type: format
+      fstype: ext3
+      label: myext3
+      volume: d2p03
+      uuid: 5d7f4d30-0c41-11e8-a664-525400123456
+    - id: fs04
+      type: format
+      fstype: ext4
+      label: myext4
+      volume: d2p04
+      uuid: 5da136b6-0c41-11e8-a664-525400123456
+    - id: fs05
+      type: format
+      fstype: fat16
+      label: myvfat16
+      volume: d2p05
+    - id: fs06
+      type: format
+      fstype: fat32
+      label: myvfat32
+      volume: d2p06
+    - id: fs07
+      type: format
+      fstype: jfs
+      label: myjfs
+      volume: d2p07
+    - id: fs08
+      type: format
+      fstype: ntfs
+      label: myntfs
+      volume: d2p08
+    - id: fs09
+      type: format
+      fstype: reiserfs
+      label: myreiserfs
+      volume: d2p09
+      uuid: 5ed8308e-0c41-11e8-a664-525400123456
+    - id: fs10
+      type: format
+      fstype: xfs
+      label: myxfs
+      volume: d2p10
+      uuid: 9c537621-f2f4-4e24-a071-e05012a1a997
diff --git a/tests/vmtests/test_fs_battery.py b/tests/vmtests/test_fs_battery.py
new file mode 100644
index 0000000..5798d48
--- /dev/null
+++ b/tests/vmtests/test_fs_battery.py
@@ -0,0 +1,179 @@
+# This file is part of curtin. See LICENSE file for copyright and license info.
+
+from . import VMBaseClass
+from .releases import base_vm_classes as relbase
+
+from curtin import config
+
+import os
+import textwrap
+
+
+def _parse_blkid_output(content):
+    """Parse the output of the 'blkid' calls in collect_script.
+
+    Input is groups of lines.  Each line is key=value. Each group
+    has the first line with key DEVNAME and last line key RESULT.
+
+    returned value is a dictionary by shortened devname like:.
+    {'part1': {'devname': 'part1', 'label': '...'}}"""
+    def _record(lines):
+        record = {}
+        for line in lines:
+            key, _, val = line.partition("=")
+            if key == 'DEVNAME':
+                bname = os.path.basename(val)
+                # bname is 'virtio-fsbattery-partX'. get just 'partX'
+                record[key.lower()] = bname.rpartition("-")[2]
+            elif key in ('RESULT', 'LABEL', 'UUID', 'TYPE'):
+                record[key.lower()] = val
+        return record
+
+    lines = []
+    records = {}
+    for line in content.splitlines():
+        lines.append(line)
+        if line.startswith("RESULT"):
+            r = _record(lines)
+            records[r['devname']] = r
+            lines = []
+
+    return records
+
+
+class TestFsBattery(VMBaseClass):
+    interactive = False
+    conf_file = "examples/tests/filesystem_battery.yaml"
+    extra_disks = ['20G']
+    collect_scripts = VMBaseClass.collect_scripts + [textwrap.dedent("""
+        cd OUTPUT_COLLECT_D
+        blkid -o export > blkid.out
+        cat /proc/mounts > proc_mounts
+        cat /proc/partitions > proc_partitions
+        find /etc/network/interfaces.d > find_interfacesd
+        cat /proc/cmdline > cmdline
+
+        set +x
+        serial="fsbattery"
+        disk=$(echo /dev/disk/by-id/*-$serial)
+        [ -b "$disk" ] || { echo "No disk with serial $serial." exit 1; }
+
+        # not all blkid versions output DEVNAME, so do it ourselves.
+        blkid -o export "$disk" | grep -q DEVNAME= &&
+           hasdev=true || hasdev=false
+        for d in $disk-part*; do
+            $hasdev || echo DEVNAME=$d
+            blkid -o export "$d"
+            echo RESULT=$?
+        done > battery-blkid
+
+        mpbase=/tmp/mp;
+        mkdir -p /tmp/mp
+        for d in $disk-part*; do
+            fstype=$(blkid -o export "$d" |
+                     awk -F= '$1 == "TYPE" { print $2 }')
+            if [ -z "$fstype" ]; then
+                msg="FAIL: blkid did not identify fstype"
+            else
+                mp="$mpbase/${d##*-}"
+                mkdir "$mp"
+                echo "${d##*-} $fstype" > "$mp.info"
+                if out=$(mount -t "$fstype" "$d" "$mp" 2>&1); then
+                    msg="PASS"
+                else
+                    rm -Rf "$mp.info" "$mp"
+                    msg="FAIL: mount $fstype failed $?: $out"
+                fi
+            fi
+            echo "${d##*-} mount: $msg"
+        done > battery-mount-umount
+
+        awk '$5 ~ mp { print $0 }' "mp=$mpbase/" \
+            /proc/1/mountinfo > battery-mountinfo
+
+        for info in $mpbase/*.info; do
+            read part fstype < "$info"
+            mp="${info%.info}"
+            out=$(umount "$mp" 2>&1) &&
+                echo "$part umount: PASS" ||
+                echo "$part umount: FAIL: $out"
+        done >> battery-mount-umount
+        """)]
+
+    def get_fs_entries(self):
+        """Return a dictionary of fs entires in config by 'partX'."""
+        stgcfg = config.load_config(self.conf_file)['storage']['config']
+        fs_entries = {}
+        for entry in stgcfg:
+            if not entry['id'].startswith("fs"):
+                continue
+            part = "part%d" % int(entry['id'][2:])
+            fs_entries[part] = entry.copy()
+        return fs_entries
+
+    def test_blkid_output(self):
+        """Check the recorded output of 'blkid -o export' on each partition.
+
+        parse parse the 'battery-blkid' collected file, and compare it
+        to expected output from reading the storage config."""
+        results = _parse_blkid_output(self.load_collect_file("battery-blkid"))
+
+        # tools for these types do not support providing uuid.
+        no_uuid_types = ['vfat', 'jfs', 'fat16', 'fat32', 'ntfs']
+        if self.release in ('trusty'):
+            no_uuid_types += ['btrfs', 'xfs']
+
+        for k, v in results.items():
+            if v['type'] in no_uuid_types:
+                del v['uuid']
+
+        # these curtin "types" show in blkid output differently.
+        type2blkid = {'fat32': 'vfat', 'fat16': 'vfat'}
+        expected = {}
+        for part, entry in self.get_fs_entries().items():
+            record = {
+                'devname': part,
+                'label': entry['label'],
+                'type': type2blkid.get(entry['fstype'], entry['fstype']),
+                'result': "0",
+            }
+            if 'uuid' in entry and record['type'] not in no_uuid_types:
+                record['uuid'] = entry['uuid']
+            expected[record['devname']] = record
+
+        self.assertEqual(expected, results)
+
+    def test_mount_umount(self):
+        """Check output of mount and unmount operations for each fs."""
+        results = self.load_collect_file("battery-mount-umount").splitlines()
+        entries = self.get_fs_entries()
+        expected = (["%s mount: PASS" % k for k in entries] +
+                    ["%s umount: PASS" % k for k in entries])
+        self.assertEqual(sorted(expected), sorted(results))
+
+
+class TrustyTestFsBattery(relbase.trusty, TestFsBattery):
+    __test__ = True
+
+
+class TrustyHWEXTestFsBattery(relbase.trusty_hwe_x, TestFsBattery):
+    __test__ = True
+
+
+class XenialGATestFsBattery(relbase.xenial_ga, TestFsBattery):
+    __test__ = True
+
+
+class XenialHWETestFsBattery(relbase.xenial_hwe, TestFsBattery):
+    __test__ = True
+
+
+class XenialEdgeTestFsBattery(relbase.xenial_edge, TestFsBattery):
+    __test__ = True
+
+
+class BionicTestFsBattery(relbase.bionic, TestFsBattery):
+    __test__ = True
+
+
+# vi: ts=4 expandtab syntax=python