Merge ~canonical-kernel-team/+git/autotest-client-tests:phlin/xfstests-loopdev into ~canonical-kernel-team/+git/autotest-client-tests:master

[Po-Hsu Lin]

Use loop device for the test, courtesy to Francis who add this code
on AWS cloud first. This patch was ported from there and rebased
on the xfstest build fix (LP: #2008049), and use /dev/loop0 for
finding which free loop device to use.

By doing so we can avoid destroying data in /dev/sdb accidentally.
Also it can reduce the hardware limit of this test, it will no
longer require a scratch drive to run but just 100G free disk space.

The required 100G disk space might be debatable here. But I didn't try out to see how much disk space we need.

The next thing we need to do is:
 * Add some limitation to this test, so that it won't be tested with smaller instances.
 * Improve the test granularity to make it easier to read / debug with
 * (Optional) Cherry-pick the test cases like what we did for ubuntu_zfs_xfs_generic test

[Sean Feole (sfeole)]

According to the XFSTests doc, the biggest disk exercised is 20GB ( This is only for BIGTEST and BIGSCRATCH tests & bigalloc_4k)
https://github.com/tytso/xfstests-bld/blob/master/Documentation/what-is-xfstests.md

I don't see a problem with lowering 100GB -> 20GB unless there is some urgent need for it.

[Francis Ginther (fginther)]

The "#DEVICE=$1" lines can be removed, otherwise this looks good. Thanks for taking this over the finish line.

This *should* work as is bare-metal hosts where we likely have a sufficiently large root disk. The cloud VMs would need work as I don't know of any instance that gives us 100GB or more by default. However, we did start down the path of having some "custom deploy" logic would is specified in the a-c-t directories and then acted upon by ckct when deploying the instance. This ran into some problems, but I think we could solve them given some more attention.

review: Approve

[Po-Hsu Lin (cypressyew)]

Applied with size reduced to 20G and the "#DEVICE=$1" lines removed.
Let's see how it goes.
Thanks!

diff --git a/ubuntu_xfstests_btrfs/control.ubuntu b/ubuntu_xfstests_btrfs/control.ubuntu
index 69fce73..78eb9b9 100644
--- a/ubuntu_xfstests_btrfs/control.ubuntu
+++ b/ubuntu_xfstests_btrfs/control.ubuntu
@@ -14,13 +14,13 @@ for uarg in args:
     key, val = uarg.split('=')
     argdict[key] = val
 
-# For right now, all of our (Ubuntu) testing makes use of a single,
-# scratch device.
+# Use a free loop device for the test, so it's less hardware dependent and safer.
+# Only use a scratch device when requested.
 #
 if 'UBUNTU_SCRATCH_DEVICE' in argdict:
     device = argdict['UBUNTU_SCRATCH_DEVICE']
 else:
-    device = '/dev/sdb'
+    device = utils.system_output('losetup -f')
 
 
 # Create all the partitions we want. These are determined by looking
@@ -38,7 +38,7 @@ else:
 os.environ['XFSTESTS_TEST_DRIVE'] = device
 
 # TEST_DEV: "device containing TEST PARTITION"
-os.environ['TEST_DEV'] = '%s1' % device
+os.environ['TEST_DEV'] = '%sp1' % device
 
 # TEST_DIR: "mount point of TEST PARTITION"
 test_mountpoint = os.path.join(job.tmpdir, 'xfstests-test')
@@ -47,7 +47,7 @@ if not os.path.isdir(test_mountpoint):
 os.environ['TEST_DIR'] = test_mountpoint
 
 # SCRATCH_DEV "device containing SCRATCH PARTITION"
-os.environ['SCRATCH_DEV'] = '%s2' % device
+os.environ['SCRATCH_DEV'] = '%sp2' % device
 
 # SCRATCH_MNT "mount point for SCRATCH PARTITION"
 # By default, we create a directory under autotest
@@ -59,7 +59,7 @@ if not os.path.isdir(scratch_mountpoint):
 os.environ['SCRATCH_MNT'] = scratch_mountpoint
 
 # SCRATCH_LOGDEV "device for scratch-fs external log"
-os.environ['SCRATCH_LOGDEV'] = '/dev/sdb3'
+os.environ['SCRATCH_LOGDEV'] = '%sp3' % device
 
 # SCRATCH_RTDEV "device for scratch-fs realtime data"
 # os.environ['SCRATCH_RTDEV'] = ''
diff --git a/ubuntu_xfstests_btrfs/create-test-partitions b/ubuntu_xfstests_btrfs/create-test-partitions
index 089bea6..67cb1ec 100644
--- a/ubuntu_xfstests_btrfs/create-test-partitions
+++ b/ubuntu_xfstests_btrfs/create-test-partitions
@@ -3,7 +3,12 @@
 
 set +x
 
-DEVICE=$1
+# Create the block_file and loop device, this would require 20G (count=20 * 1G)
+# disk space as suggested in what-is-xfstests.md from XFSTests doc
+sudo dd if=/dev/zero of=/block_file bs=1G count=20
+sudo losetup -fP /block_file
+DEVICE=$(sudo losetup -a|grep block_file|cut -d":" -f1)
+
 FS=$2
 
 if [ -z $DEVICE ]; then
@@ -48,11 +53,12 @@ fi
 
 sleep 60 # Give things a minute to settle before creating new partitions
 
+ORIG_DEVICE=${DEVICE}
 [[ $DEVICE =~ ^/dev/loop[0-9]{1,}$ ]] && DEVICE=${DEVICE}p
 
 for i in `seq 1 5`
 do
     sudo mkfs.$FS $FS_OPT ${DEVICE}$i
 done
-sudo parted $DEVICE print
+sudo parted $ORIG_DEVICE print
 
diff --git a/ubuntu_xfstests_ext4/control.ubuntu b/ubuntu_xfstests_ext4/control.ubuntu
index f81ecde..4561ced 100644
--- a/ubuntu_xfstests_ext4/control.ubuntu
+++ b/ubuntu_xfstests_ext4/control.ubuntu
@@ -14,13 +14,13 @@ for uarg in args:
     key, val = uarg.split('=')
     argdict[key] = val
 
-# For right now, all of our (Ubuntu) testing makes use of a single,
-# scratch device.
+# Use a free loop device for the test, so it's less hardware dependent and safer.
+# Only use a scratch device when requested.
 #
 if 'UBUNTU_SCRATCH_DEVICE' in argdict:
     device = argdict['UBUNTU_SCRATCH_DEVICE']
 else:
-    device = '/dev/sdb'
+    device = utils.system_output('losetup -f')
 
 
 # Create all the partitions we want. These are determined by looking
@@ -38,7 +38,7 @@ else:
 os.environ['XFSTESTS_TEST_DRIVE'] = device
 
 # TEST_DEV: "device containing TEST PARTITION"
-os.environ['TEST_DEV'] = '%s1' % device
+os.environ['TEST_DEV'] = '%sp1' % device
 
 # TEST_DIR: "mount point of TEST PARTITION"
 test_mountpoint = os.path.join(job.tmpdir, 'xfstests-test')
@@ -47,7 +47,7 @@ if not os.path.isdir(test_mountpoint):
 os.environ['TEST_DIR'] = test_mountpoint
 
 # SCRATCH_DEV "device containing SCRATCH PARTITION"
-os.environ['SCRATCH_DEV'] = '%s2' % device
+os.environ['SCRATCH_DEV'] = '%sp2' % device
 
 # SCRATCH_MNT "mount point for SCRATCH PARTITION"
 # By default, we create a directory under autotest
@@ -59,7 +59,7 @@ if not os.path.isdir(scratch_mountpoint):
 os.environ['SCRATCH_MNT'] = scratch_mountpoint
 
 # SCRATCH_LOGDEV "device for scratch-fs external log"
-os.environ['SCRATCH_LOGDEV'] = '/dev/sdb3'
+os.environ['SCRATCH_LOGDEV'] = '%sp3' % device
 
 # SCRATCH_RTDEV "device for scratch-fs realtime data"
 # os.environ['SCRATCH_RTDEV'] = ''
diff --git a/ubuntu_xfstests_ext4/create-test-partitions b/ubuntu_xfstests_ext4/create-test-partitions
index 089bea6..67cb1ec 100644
--- a/ubuntu_xfstests_ext4/create-test-partitions
+++ b/ubuntu_xfstests_ext4/create-test-partitions
@@ -3,7 +3,12 @@
 
 set +x
 
-DEVICE=$1
+# Create the block_file and loop device, this would require 20G (count=20 * 1G)
+# disk space as suggested in what-is-xfstests.md from XFSTests doc
+sudo dd if=/dev/zero of=/block_file bs=1G count=20
+sudo losetup -fP /block_file
+DEVICE=$(sudo losetup -a|grep block_file|cut -d":" -f1)
+
 FS=$2
 
 if [ -z $DEVICE ]; then
@@ -48,11 +53,12 @@ fi
 
 sleep 60 # Give things a minute to settle before creating new partitions
 
+ORIG_DEVICE=${DEVICE}
 [[ $DEVICE =~ ^/dev/loop[0-9]{1,}$ ]] && DEVICE=${DEVICE}p
 
 for i in `seq 1 5`
 do
     sudo mkfs.$FS $FS_OPT ${DEVICE}$i
 done
-sudo parted $DEVICE print
+sudo parted $ORIG_DEVICE print
 
diff --git a/ubuntu_xfstests_xfs/control.ubuntu b/ubuntu_xfstests_xfs/control.ubuntu
index 0cfad11..4e2c7ea 100644
--- a/ubuntu_xfstests_xfs/control.ubuntu
+++ b/ubuntu_xfstests_xfs/control.ubuntu
@@ -14,13 +14,13 @@ for uarg in args:
     key, val = uarg.split('=')
     argdict[key] = val
 
-# For right now, all of our (Ubuntu) testing makes use of a single,
-# scratch device.
+# Use a free loop device for the test, so it's less hardware dependent and safer.
+# Only use a scratch device when requested.
 #
 if 'UBUNTU_SCRATCH_DEVICE' in argdict:
     device = argdict['UBUNTU_SCRATCH_DEVICE']
 else:
-    device = '/dev/sdb'
+    device = utils.system_output('losetup -f')
 
 
 # Create all the partitions we want. These are determined by looking
@@ -38,7 +38,7 @@ else:
 os.environ['XFSTESTS_TEST_DRIVE'] = device
 
 # TEST_DEV: "device containing TEST PARTITION"
-os.environ['TEST_DEV'] = '%s1' % device
+os.environ['TEST_DEV'] = '%sp1' % device
 
 # TEST_DIR: "mount point of TEST PARTITION"
 test_mountpoint = os.path.join(job.tmpdir, 'xfstests-test')
@@ -47,7 +47,7 @@ if not os.path.isdir(test_mountpoint):
 os.environ['TEST_DIR'] = test_mountpoint
 
 # SCRATCH_DEV "device containing SCRATCH PARTITION"
-os.environ['SCRATCH_DEV'] = '%s2' % device
+os.environ['SCRATCH_DEV'] = '%sp2' % device
 
 # SCRATCH_MNT "mount point for SCRATCH PARTITION"
 # By default, we create a directory under autotest
@@ -59,7 +59,7 @@ if not os.path.isdir(scratch_mountpoint):
 os.environ['SCRATCH_MNT'] = scratch_mountpoint
 
 # SCRATCH_LOGDEV "device for scratch-fs external log"
-os.environ['SCRATCH_LOGDEV'] = '/dev/sdb3'
+os.environ['SCRATCH_LOGDEV'] = '%sp3' % device
 
 # SCRATCH_RTDEV "device for scratch-fs realtime data"
 # os.environ['SCRATCH_RTDEV'] = ''
diff --git a/ubuntu_xfstests_xfs/create-test-partitions b/ubuntu_xfstests_xfs/create-test-partitions
index 089bea6..67cb1ec 100644
--- a/ubuntu_xfstests_xfs/create-test-partitions
+++ b/ubuntu_xfstests_xfs/create-test-partitions
@@ -3,7 +3,12 @@
 
 set +x
 
-DEVICE=$1
+# Create the block_file and loop device, this would require 20G (count=20 * 1G)
+# disk space as suggested in what-is-xfstests.md from XFSTests doc
+sudo dd if=/dev/zero of=/block_file bs=1G count=20
+sudo losetup -fP /block_file
+DEVICE=$(sudo losetup -a|grep block_file|cut -d":" -f1)
+
 FS=$2
 
 if [ -z $DEVICE ]; then
@@ -48,11 +53,12 @@ fi
 
 sleep 60 # Give things a minute to settle before creating new partitions
 
+ORIG_DEVICE=${DEVICE}
 [[ $DEVICE =~ ^/dev/loop[0-9]{1,}$ ]] && DEVICE=${DEVICE}p
 
 for i in `seq 1 5`
 do
     sudo mkfs.$FS $FS_OPT ${DEVICE}$i
 done
-sudo parted $DEVICE print
+sudo parted $ORIG_DEVICE print