Merge lp:~raharper/curtin/trunk.zesty-btrfs into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

vmtest: handle change in btrfs-progs when finding fs UUID.

Refactor how we extract the btrfs filesystem UUID with btrfs-progs tools.

[lp:~raharper/curtin/trunk.zesty-btrfs]

470. By Ryan Harper on 2017-02-22

Drop btrfs collect from nvme test

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:469
https://jenkins.ubuntu.com/server/job/curtin-ci/385/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/385
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/385
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/385
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-amd64/385
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/385

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/385/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:470
https://jenkins.ubuntu.com/server/job/curtin-ci/386/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/386
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/386
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/386
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-amd64/386
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/386

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/386/rebuild

review: Approve (continuous-integration)

[Joshua Powers (powersj)]

review: Approve

[Scott Moser (smoser)]

I'm fine with this, but it might make more sense to make the "collector" smarter than having 2 different collectors (btrfs-debug-tree and btrfs-show ). What I'm suggesting is basically trying to produce a single 'btrfs-uuids' file with something like:

for dev in $(find_btrfs_devs); do
   if command -v btrfs-show >/dev/null 2>&1; then
      btrfs-show $dev | awk '/uuid/ {print dev, $4}' dev=$dev
   elif command -v btrfs-debug-tree >/dev/null 2>&1; then
      btrfs-debug-tree -r $dev | awk '/^uuid/ {print dev, $2}' dev=$dev | grep "-"
   fi

done > btrfs-uuids

I realize that is more work on your part, and that this is breaking vmtest, but I think generally we want to get to generic "collect a lot of stuff in the guest and then look at the results".

I'll approve, though. but at some point we are going to have to re-factor our inspection code.

review: Approve

[Ryan Harper (raharper)]

On Wed, Feb 22, 2017 at 8:30 AM, Scott Moser <email address hidden> wrote:

> Review: Approve
>
> I'm fine with this, but it might make more sense to make the "collector"
> smarter than having 2 different collectors (btrfs-debug-tree and
> btrfs-show ). What I'm suggesting is basically trying to produce a single
> 'btrfs-uuids' file with something like:
>
> for dev in $(find_btrfs_devs); do
> if command -v btrfs-show >/dev/null 2>&1; then
> btrfs-show $dev | awk '/uuid/ {print dev, $4}' dev=$dev
> elif command -v btrfs-debug-tree >/dev/null 2>&1; then
> btrfs-debug-tree -r $dev | awk '/^uuid/ {print dev, $2}' dev=$dev |
> grep "-"
> fi
>
> done > btrfs-uuids
>
> I realize that is more work on your part, and that this is breaking
> vmtest, but I think generally we want to get to generic "collect a lot of
> stuff in the guest and then look at the results".
>

So, it's precise *only* which will fall out in April this year which
doesn't have the now even more common btrfs-debug-tree.

>
> I'll approve, though. but at some point we are going to have to re-factor
> our inspection code.
>

Yes, the trunk.power8 branch does much of this so we're not repeating these
scripts in N test-files; and in that case
I agree it makes more sense to be release-tolerant in the collector.

>
> --
> https://code.launchpad.net/~raharper/curtin/trunk.zesty-
> btrfs/+merge/317931
> You are the owner of lp:~raharper/curtin/trunk.zesty-btrfs.
>

=== modified file 'tests/vmtests/test_basic.py'
--- tests/vmtests/test_basic.py	2017-02-08 17:50:26 +0000
+++ tests/vmtests/test_basic.py	2017-02-22 01:42:33 +0000
@@ -3,7 +3,6 @@
     get_apt_proxy)
 from .releases import base_vm_classes as relbase
 
-import re
 import textwrap
 
 
@@ -19,7 +18,8 @@
         blkid -o export /dev/vda > blkid_output_vda
         blkid -o export /dev/vda1 > blkid_output_vda1
         blkid -o export /dev/vda2 > blkid_output_vda2
-        btrfs-show-super /dev/vdd > btrfs_show_super_vdd
+        f="btrfs_uuid_vdd"
+        btrfs-debug-tree -r /dev/vdd | awk '/^uuid/ {print $2}' | grep "-" > $f
         cat /proc/partitions > proc_partitions
         ls -al /dev/disk/by-uuid/ > ls_uuid
         cat /etc/fstab > fstab
@@ -36,7 +36,7 @@
     def test_output_files_exist(self):
         self.output_files_exist(
             ["blkid_output_vda", "blkid_output_vda1", "blkid_output_vda2",
-             "btrfs_show_super_vdd", "fstab", "ls_dname", "ls_uuid",
+             "btrfs_uuid_vdd", "fstab", "ls_dname", "ls_uuid",
              "proc_partitions"])
 
     def test_ptable(self):
@@ -89,15 +89,12 @@
 
     def test_whole_disk_format(self):
         # confirm the whole disk format is the expected device
-        btrfs_show_super = self.load_collect_file('btrfs_show_super_vdd')
+        btrfs_uuid = self.load_collect_file('btrfs_uuid_vdd').strip()
         ls_uuid = self.load_collect_file("ls_uuid")
 
         # extract uuid from btrfs superblock
-        btrfs_fsid = [line for line in btrfs_show_super.split('\n')
-                      if line.startswith('fsid\t\t')]
-        self.assertEqual(len(btrfs_fsid), 1)
-        btrfs_uuid = btrfs_fsid[0].split()[1]
         self.assertTrue(btrfs_uuid is not None)
+        self.assertEqual(len(btrfs_uuid), 36)
 
         # extract uuid from /dev/disk/by-uuid on /dev/vdd
         # parsing ls -al output on /dev/disk/by-uuid:
@@ -138,7 +135,8 @@
         blkid -o export /dev/vda > blkid_output_vda
         blkid -o export /dev/vda1 > blkid_output_vda1
         blkid -o export /dev/vda2 > blkid_output_vda2
-        btrfs-show /dev/vdd > btrfs_show_super_vdd
+        f="btrfs_uuid_vdd"
+        btrfs-show /dev/vdd | awk '/uuid/ {print $4}' > $f
         cat /proc/partitions > proc_partitions
         ls -al /dev/disk/by-uuid/ > ls_uuid
         cat /etc/fstab > fstab
@@ -154,15 +152,11 @@
 
     def test_whole_disk_format(self):
         # confirm the whole disk format is the expected device
-        btrfs_show_super = self.load_collect_file("btrfs_show_super_vdd")
+        btrfs_uuid = self.load_collect_file("btrfs_uuid_vdd").strip()
         ls_uuid = self.load_collect_file("ls_uuid")
 
-        # extract uuid from btrfs superblock
-        btrfs_fsid = re.findall('.*uuid:\ (.*)\n', btrfs_show_super)
-
-        self.assertEqual(len(btrfs_fsid), 1)
-        btrfs_uuid = btrfs_fsid.pop()
         self.assertTrue(btrfs_uuid is not None)
+        self.assertEqual(len(btrfs_uuid), 36)
 
         # extract uuid from /dev/disk/by-uuid on /dev/vdd
         # parsing ls -al output on /dev/disk/by-uuid:
@@ -233,7 +227,8 @@
         blkid -o export /dev/sda > blkid_output_sda
         blkid -o export /dev/sda1 > blkid_output_sda1
         blkid -o export /dev/sda2 > blkid_output_sda2
-        btrfs-show-super /dev/sdc > btrfs_show_super_sdc
+        f="btrfs_uuid_sdc"
+        btrfs-debug-tree -r /dev/sdc | awk '/^uuid/ {print $2}' | grep "-" > $f
         cat /proc/partitions > proc_partitions
         ls -al /dev/disk/by-uuid/ > ls_uuid
         ls -al /dev/disk/by-id/ > ls_disk_id
@@ -251,7 +246,7 @@
     def test_output_files_exist(self):
         self.output_files_exist(
             ["blkid_output_sda", "blkid_output_sda1", "blkid_output_sda2",
-             "btrfs_show_super_sdc", "fstab", "ls_dname", "ls_uuid",
+             "btrfs_uuid_sdc", "fstab", "ls_dname", "ls_uuid",
              "ls_disk_id", "proc_partitions"])
 
     def test_ptable(self):
@@ -304,15 +299,12 @@
 
     def test_whole_disk_format(self):
         # confirm the whole disk format is the expected device
-        btrfs_show_super = self.load_collect_file("btrfs_show_super_sdc")
+        btrfs_uuid = self.load_collect_file("btrfs_uuid_sdc").strip()
         ls_uuid = self.load_collect_file("ls_uuid")
 
         # extract uuid from btrfs superblock
-        btrfs_fsid = [line for line in btrfs_show_super.split('\n')
-                      if line.startswith('fsid\t\t')]
-        self.assertEqual(len(btrfs_fsid), 1)
-        btrfs_uuid = btrfs_fsid[0].split()[1]
         self.assertTrue(btrfs_uuid is not None)
+        self.assertEqual(len(btrfs_uuid), 36)
 
         # extract uuid from /dev/disk/by-uuid on /dev/sdc
         # parsing ls -al output on /dev/disk/by-uuid:

=== modified file 'tests/vmtests/test_nvme.py'
--- tests/vmtests/test_nvme.py	2017-02-01 20:32:13 +0000
+++ tests/vmtests/test_nvme.py	2017-02-22 01:42:33 +0000
@@ -25,7 +25,6 @@
         blkid -o export /dev/vda > blkid_output_vda
         blkid -o export /dev/vda1 > blkid_output_vda1
         blkid -o export /dev/vda2 > blkid_output_vda2
-        btrfs-show-super /dev/vdd > btrfs_show_super_vdd
         cat /proc/partitions > proc_partitions
         ls -al /dev/disk/by-uuid/ > ls_uuid
         cat /etc/fstab > fstab