Merge ~pieq/plainbox-provider-checkbox:1660386-mount-drive-before-storage-tests into plainbox-provider-checkbox:master

[Pierre Equoy]

Automatically mount drives when running storage tests

When running disk/storage_device_* jobs, if the drive is not mounted, we now check if it can be mounted (i.e. if it has a partition table) and if possible we mount it to a dedicated temporary location in order to perform the storage tests.

To test this, run checkbox-cli with a test plan containing the appropriate jobs (
disk/storage_device_*).

Sample output:

---------------[ Disk I/O stress test for HGST HTS541010A7E630 ]----------------
ID: 2013.com.canonical.certification::disk/storage_device_sda
Category: 2013.com.canonical.plainbox::disk
... 8< -------------------------------------------------------------------------
/dev/sda is a block device
/dev/sda reports a size of 1000GB.
/dev/sda is not mounted. It must be mounted before testing.
Mounting /dev/sda into /tmp/drive/sda...
Running bonnie++ on /dev/sda...
Putting scratch disk at /tmp/drive/sda
    Disk /dev/sda has 890992MB available
    System has 15914MB RAM
    We need 31828MB of disk space for testing
   Free disk space is sufficient to continue testing.
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
u-Unidentifi 31824M 1309 96 108464 5 54533 5 4850 90 140292 7 251.4 4
Latency 51116us 1556ms 7425ms 24574us 51127us 690ms
Version 1.97 ------Sequential Create------ --------Random Create--------
u-Unidentified-Syst -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 343us 559us 591us 249us 71us 779us
1.97,1.97,u-Unidentified-System,1,1487159060,31824M,,1309,96,108464,5,54533,5,4850,90,140292,7,251.4,4,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,51116us,1556ms,7425ms,24574us,51127us,690ms,343us,559us,591us,249us,71us,779us
------------------------------------------------------------------------- >8 ---
Outcome: job passed

[Jeff Lane  (bladernr)]

So just for comparison, this is what happens when you run this against a disk with no partition table:
ubuntu@wildorange:~$ sudo ./storage_test sdc
/dev/sdc is a block device
Error: /dev/sdc: unrecognised disk label
Error: /dev/sdb: unrecognised disk label
Error: /dev/sdc: unrecognised disk label
Error: /dev/sdd: unrecognised disk label
/dev/sdc reports a size of 147GB.
/dev/sdc is not mounted. It must be mounted before testing.
Error: /dev/sdc: unrecognised disk label
/dev/sdc has no partition table. Test on this drive cannot proceed.

There's a lot of stderr there that makes reading the output confusing...

[Jeff Lane  (bladernr)]

This is running against a disk that has a disk label, but no partitions:
ubuntu@wildorange:~$ sudo ./storage_test sdb
/dev/sdb is a block device
/dev/sdb reports a size of 147GB.
/dev/sdb is not mounted. It must be mounted before testing.
Mounting /dev/sdb into /tmp/drive/sdb...
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Running bonnie++ on /dev/sdb...
Putting scratch disk at
    Disk /dev/sdb has MB available
    System has 56354MB RAM
    We need 112708MB of disk space for testing
    Insufficient disk space available for defaults.
    reducing memory footprint to be 1/2 of free disk space.
./storage_test: line 25: /2: syntax error: operand expected (error token is "/2")

Completely different error there...

[Jeff Lane  (bladernr)]

This is running against a disk with a label and a partition, but the partition is not formatted:
ubuntu@wildorange:~$ sudo ./storage_test sdc
/dev/sdc is a block device
/dev/sdc reports a size of 147GB.
/dev/sdc is not mounted. It must be mounted before testing.
Mounting /dev/sdc into /tmp/drive/sdc...
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Running bonnie++ on /dev/sdc...
Putting scratch disk at
    Disk /dev/sdc has MB available
    System has 56354MB RAM
    We need 112708MB of disk space for testing
    Insufficient disk space available for defaults.
    reducing memory footprint to be 1/2 of free disk space.
./storage_test: line 25: /2: syntax error: operand expected (error token is "/2")

Same error at line 25, so that should probably be fixed.

[Jeff Lane  (bladernr)]

And disk with label, and 1 partition formatted ext4:
ubuntu@wildorange:~$ sudo mkfs.ext4 /dev/sdd1
mke2fs 1.42.13 (17-May-2015)
/dev/sdd1 contains a ext4 file system
 created on Wed Feb 15 15:27:09 2017
Proceed anyway? (y,n) y
Creating filesystem with 14648064 4k blocks and 3662848 inodes
Filesystem UUID: d6f65789-4702-4f9e-be5b-a76aeceb11cd
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
 4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

ubuntu@wildorange:~$ sudo ./storage_test sdd
/dev/sdd is a block device
/dev/sdd reports a size of 147GB.
/dev/sdd is not mounted. It must be mounted before testing.
Mounting /dev/sdd into /tmp/drive/sdd...
mount: wrong fs type, bad option, bad superblock on /dev/sdd,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Running bonnie++ on /dev/sdd...
Putting scratch disk at
    Disk /dev/sdd has MB available
    System has 56354MB RAM
    We need 112708MB of disk space for testing
    Insufficient disk space available for defaults.
    reducing memory footprint to be 1/2 of free disk space.
./storage_test: line 25: /2: syntax error: operand expected (error token is "/2")

this should have worked, but didn't.

[Jeff Lane  (bladernr)]

Now, I have a disk with 2 partitions, one is 60Gb ext4 and the other is roughly 80ish GB ext3.
ubuntu@wildorange:~$ sudo parted /dev/sdd
GNU Parted 3.2
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) q
ubuntu@wildorange:~$ sudo ./storage_test sdd
/dev/sdd is a block device
/dev/sdd reports a size of 147GB.
/dev/sdd is not mounted. It must be mounted before testing.
Mounting /dev/sdd into /tmp/drive/sdd...
mount: wrong fs type, bad option, bad superblock on /dev/sdd,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Running bonnie++ on /dev/sdd...
Putting scratch disk at
    Disk /dev/sdd has MB available
    System has 56354MB RAM
    We need 112708MB of disk space for testing
    Insufficient disk space available for defaults.
    reducing memory footprint to be 1/2 of free disk space.
./storage_test: line 25: /2: syntax error: operand expected (error token is "/2")

same output as above

[Jeff Lane  (bladernr)]

So after looking a bit I realized what is happening and why it's broken. Comment inline. I think on your test system, since you only have one drive and it was already mounted (rootfs), you wouldn't have run into the issue that I have on a system with 4 disks.

review: Needs Fixing

[Pierre Equoy (pieq)]

Thanks a lot for your comments, Jeff!

I adapted a little bit the find_largest_partition (we use this test on laptop/desktops, so no need to check for LVM on unmounted drives I think).

I also got rid of the partition table check using parted, as it became useless when using find_largest_partition. If there is (largest) partition, we're all good, if not we just display an error and some help.

[Pierre Equoy (pieq)]

[Re-submit]

review: Needs Resubmitting

[Jeff Lane  (bladernr)]

Cool, Looks good and I retried on all the previous test cases that failed, and they all pass now (or at least they don't throw nasty errors and such), they do what I'd expect them to do in each case.

review: Approve

diff --git a/bin/storage_test b/bin/storage_test
index ef19c5e..40cc3f2 100755
--- a/bin/storage_test
+++ b/bin/storage_test
@@ -30,6 +30,34 @@ function run_bonnie() {
     fi
 }
 
+# Find the largest partition that holds a supported filesystem on $disk_device.
+# This code is adapted from a similar function in `disk_stress_ng`.
+# Output:
+#   $largest_part -- Device filename of largest qualifying partition or logical volume
+#   $largest_size -- Size of largest qualifying partition or logical volume
+#   $largest_fs -- Filesystem (ext4, etc.) used on largest qualifying partition or logical volume
+#   $unsupported_fs -- Empty or contains name of unsupported filesystem found on disk
+find_largest_partition() {
+    largest_part=""
+    largest_size=0
+    partitions=$(lsblk -b -l -n -o NAME,SIZE,TYPE,MOUNTPOINT $disk | grep part | tr -s " ")
+    unsupported_fs=""
+    for partition in $(echo "$partitions" | cut -d " " -f 1) ; do
+        part_size=$(echo "$partitions" | grep "$partition " | cut -d " " -f 2)
+        local blkid_info=$(blkid -s TYPE /dev/$partition | grep -E ext2\|ext3\|ext4\|xfs\|jfs\|btrfs)
+        if [ "$part_size" -gt "$largest_size" ] && [ -n "$blkid_info" ]  ; then
+            largest_size=$part_size
+            largest_part="/dev/$partition"
+            largest_fs=$(blkid -s TYPE "/dev/$partition" | cut -d "=" -f 2)
+        fi
+        local blkid_info=$(blkid -s TYPE /dev/$partition | grep -E ntfs\|vfat\|hfs)
+        if [ -n "$blkid_info" ]  ; then
+            # If there's an NTFS, HFS+, or FAT filesystem on the disk make note of it....
+            unsupported_fs=$(blkid -s TYPE "/dev/$partition" | cut -d "=" -f 2)
+        fi
+    done
+} # find_largest_partition()
+
 disk=/dev/$1
 
 if [ -b $disk ]
@@ -62,7 +90,17 @@ then
             echo "$disk is mounted, proceeding."
         else
             echo "$disk is not mounted. It must be mounted before testing."
-            exit 1
+            find_largest_partition
+            if [ -n "$largest_part" ]
+            then
+                dest=/tmp/drive/$partition
+                echo "Mounting $largest_part into $dest..."
+                mkdir -p $dest
+                mount $largest_part $dest
+            else
+                echo "$disk has no partition. Please format this drive and re-launch the test."
+                exit 1
+            fi
         fi