Merge lp:~raharper/curtin/trunk.dirty-disks-in-early-command into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

vmtests: enable tests apply storage config to validate clear_holders path

Allow test classes with storage configurations to optionally run curtin
block-meta during the early stage to "dirty" the disks used in the
test-case. This exercises curtin's clear-holders path to closer match
MAAS redeploying systems with existing storage configurations. In some
cases, we need to run with multiple virtual processors to allow race
conditions to occur and exercise curtin's handling of these situations.

- Add class attribute 'nr_cpus' (integer) to control number of VCPUS used
  in the guest.
  Defaults to 1, allows CURTIN_VMTEST_NR_CPUS env to override for all tests.
- Add class attribute 'dirty_disks' (boolean) to control if a test class
  will enable running the early command.
  Defaults to False
- Add class attribute 'dirty_disk_config' (string) which is path to the
  generic early_command yaml blob used to run curtin block-meta with the
  test-case's storage config.
  Defaults to 'examples/tests/dirty_disks_config.yaml'
- tools/launch accepts --smp to control the number of VCPUS to use in the
  VM

[lp:~raharper/curtin/trunk.dirty-disks-in-early-command]

498. By Ryan Harper on 2017-04-25

Remove early_commands from bcache, enable via class attr, whitespace

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:498
https://jenkins.ubuntu.com/server/job/curtin-ci/478/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/478
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/478
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/478
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/478
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/478

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/478/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

review: Approve

[lp:~raharper/curtin/trunk.dirty-disks-in-early-command]

499. By Ryan Harper on 2017-04-26

merge from trunk

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:499
https://jenkins.ubuntu.com/server/job/curtin-ci/483/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/483
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/483
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/483
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/483
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/483

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/483/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.dirty-disks-in-early-command]

500. By Ryan Harper on 2017-04-26

Add wipe: superblock to all storage configs

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:500
https://jenkins.ubuntu.com/server/job/curtin-ci/484/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/484
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/484
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/484
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/484
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/484

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/484/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.dirty-disks-in-early-command]

501. By Ryan Harper on 2017-04-26

Fix mising comma after smp entry in getops long_opts

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:501
https://jenkins.ubuntu.com/server/job/curtin-ci/485/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/485
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/485
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/485
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/485
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/485

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/485/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.dirty-disks-in-early-command]

502. By Ryan Harper on 2017-05-03

merge from trunk

503. By Ryan Harper on 2017-05-03

Use nr_cpus and dirty_disks to handle regression testing of mirrorboot-uefi

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:503
https://jenkins.ubuntu.com/server/job/curtin-ci/487/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/487
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/487
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/487
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/487

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/487/rebuild

review: Approve (continuous-integration)

[lp:~raharper/curtin/trunk.dirty-disks-in-early-command]

504. By Ryan Harper on 2017-05-15

merge from trunk

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:504
https://jenkins.ubuntu.com/server/job/curtin-ci/505/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/505
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/505
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/505
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=vm-i386/505

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/505/rebuild

review: Approve (continuous-integration)

=== modified file 'doc/topics/integration-testing.rst'
--- doc/topics/integration-testing.rst	2017-04-21 18:09:54 +0000
+++ doc/topics/integration-testing.rst	2017-05-15 15:58:10 +0000
@@ -205,6 +205,11 @@
     This modifies the  invocation of nosetets to add '--processes' and other
     necessary nose arguments (--process-timeout)
 
+- ``CURTIN_VMTEST_NR_CPUS``: default ''
+
+    Allow environment to override the number of virtual cpus to allocate
+    in the target virtual machines.
+
 - ``IMAGE_DIR``: default /srv/images
 
   Vmtest keeps a mirror of maas ephemeral images in this directory.

=== modified file 'examples/tests/allindata.yaml'
--- examples/tests/allindata.yaml	2016-04-14 01:45:54 +0000
+++ examples/tests/allindata.yaml	2017-05-15 15:58:10 +0000
@@ -4,6 +4,7 @@
   config:
      - id: sda
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdb
@@ -42,6 +43,7 @@
        number: 6  # XXX: we really need to stop using id with DiskPartnum
      - id: sdb
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdc
@@ -64,6 +66,7 @@
        device: sdb
      - id: sdc
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdd
@@ -86,6 +89,7 @@
        device: sdc
      - id: sdd
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vde

=== modified file 'examples/tests/basic.yaml'
--- examples/tests/basic.yaml	2016-08-01 17:18:18 +0000
+++ examples/tests/basic.yaml	2017-05-15 15:58:10 +0000
@@ -25,6 +25,7 @@
         type: format
         fstype: ext4
         volume: sda1
+        label: 'cloudimg-rootfs'
       - id: sda2_home
         type: format
         fstype: ext4

=== modified file 'examples/tests/bcache_basic.yaml'
--- examples/tests/bcache_basic.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/bcache_basic.yaml	2017-05-15 15:58:10 +0000
@@ -1,4 +1,5 @@
 showtrace: true
+
 storage:
   config:
   - id: id_rotary0

=== added file 'examples/tests/dirty_disks_config.yaml'
--- examples/tests/dirty_disks_config.yaml	1970-01-01 00:00:00 +0000
+++ examples/tests/dirty_disks_config.yaml	2017-05-15 15:58:10 +0000
@@ -0,0 +1,10 @@
+early_commands:
+  # running block-meta custom from the install environment
+  # inherits the CONFIG environment, so this works to actually prepare
+  # the disks exactly as in this config before the rest of the install
+  # will just blow it all away.  We have clean out other environment
+  # that could unintentionally mess things up.
+  blockmeta: [env, -u, OUTPUT_FSTAB,
+              TARGET_MOUNT_POINT=/tmp/my.bdir/target,
+              WORKING_DIR=/tmp/my.bdir/work.d, 
+              curtin, --showtrace, -v, block-meta, --umount, custom]

=== modified file 'examples/tests/lvm.yaml'
--- examples/tests/lvm.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/lvm.yaml	2017-05-15 15:58:10 +0000
@@ -4,6 +4,7 @@
     config:
       - id: sda
         type: disk
+        wipe: superblock
         ptable: msdos
         model: QEMU HARDDISK
         path: /dev/vdb

=== modified file 'examples/tests/mdadm_bcache.yaml'
--- examples/tests/mdadm_bcache.yaml	2016-04-17 22:35:20 +0000
+++ examples/tests/mdadm_bcache.yaml	2017-05-15 15:58:10 +0000
@@ -5,6 +5,7 @@
      - grub_device: true
        id: sda
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdb
@@ -53,11 +54,13 @@
        uuid: deadbeef-dead-beef-dead-deadbeefaaa6
      - id: sdb
        type: disk
+       wipe: superblock
        model: QEMU HARDDISK
        path: /dev/vdc
        name: second_disk
      - id: sdc
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdd

=== modified file 'examples/tests/mdadm_bcache_complex.yaml'
--- examples/tests/mdadm_bcache_complex.yaml	2015-11-25 13:45:53 +0000
+++ examples/tests/mdadm_bcache_complex.yaml	2017-05-15 15:58:10 +0000
@@ -4,6 +4,7 @@
      - grub_device: true
        id: sda
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdb
@@ -43,11 +44,13 @@
        device: sda
      - id: sdb
        type: disk
+       wipe: superblock
        model: QEMU HARDDISK
        path: /dev/vdc
        name: second_disk
      - id: sdc
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdd

=== modified file 'examples/tests/mirrorboot-uefi.yaml'
--- examples/tests/mirrorboot-uefi.yaml	2017-05-03 02:10:47 +0000
+++ examples/tests/mirrorboot-uefi.yaml	2017-05-15 15:58:10 +0000
@@ -1,16 +1,4 @@
 showtrace: true
-
-early_commands:
-  # running block-meta custom from the install environment
-  # inherits the CONFIG environment, so this works to actually prepare
-  # the disks exactly as in this config before the rest of the install
-  # will just blow it all away.  We have to clean out the other
-  # environment that could unintentionally mess things up.
-  blockmeta: [env, -u, OUTPUT_FSTAB,
-              TARGET_MOUNT_POINT=/tmp/my.bdir/target,
-              WORKING_DIR=/tmp/my.bdir/work.d, 
-              curtin, --showtrace, -v, block-meta, --umount, custom]
-
 storage:
   config:
   - grub_device: true

=== modified file 'examples/tests/mirrorboot.yaml'
--- examples/tests/mirrorboot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/mirrorboot.yaml	2017-05-15 15:58:10 +0000
@@ -4,6 +4,7 @@
   config:
      - id: sda
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdb
@@ -20,6 +21,7 @@
        device: sda
      - id: sdb
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdc

=== modified file 'examples/tests/raid10boot.yaml'
--- examples/tests/raid10boot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/raid10boot.yaml	2017-05-15 15:58:10 +0000
@@ -4,6 +4,7 @@
   config:
      - id: sda
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdb
@@ -20,6 +21,7 @@
        device: sda
      - id: sdb
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdc
@@ -30,6 +32,7 @@
        device: sdb
      - id: sdc
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdd
@@ -40,6 +43,7 @@
        device: sdc
      - id: sdd
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vde

=== modified file 'examples/tests/raid5boot.yaml'
--- examples/tests/raid5boot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/raid5boot.yaml	2017-05-15 15:58:10 +0000
@@ -4,6 +4,7 @@
   config:
      - id: sda
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdb
@@ -20,6 +21,7 @@
        device: sda
      - id: sdb
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdc
@@ -30,6 +32,7 @@
        device: sdb
      - id: sdc
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdd

=== modified file 'examples/tests/raid6boot.yaml'
--- examples/tests/raid6boot.yaml	2016-04-04 16:03:32 +0000
+++ examples/tests/raid6boot.yaml	2017-05-15 15:58:10 +0000
@@ -4,6 +4,7 @@
   config:
      - id: sda
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdb
@@ -20,6 +21,7 @@
        device: sda
      - id: sdb
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdc
@@ -30,6 +32,7 @@
        device: sdb
      - id: sdc
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vdd
@@ -40,6 +43,7 @@
        device: sdc
      - id: sdd
        type: disk
+       wipe: superblock
        ptable: gpt
        model: QEMU HARDDISK
        path: /dev/vde

=== modified file 'tests/vmtests/__init__.py'
--- tests/vmtests/__init__.py	2017-04-21 20:08:30 +0000
+++ tests/vmtests/__init__.py	2017-05-15 15:58:10 +0000
@@ -333,6 +333,9 @@
     boot_timeout = BOOT_TIMEOUT
     collect_scripts = []
     conf_file = "examples/tests/basic.yaml"
+    nr_cpus = None
+    dirty_disks = False
+    dirty_disk_config = "examples/tests/dirty_disks_config.yaml"
     disk_block_size = 512
     disk_driver = 'virtio-blk'
     disk_to_check = {}
@@ -507,6 +510,26 @@
                 bugnum, clsname)
 
     @classmethod
+    def get_config_smp(cls):
+        """Get number of cpus to use for guest"""
+
+        nr_cpus = None
+        if cls.nr_cpus:
+            nr_cpus = cls.nr_cpus
+            logger.debug('Setting cpus from class value: %s', nr_cpus)
+
+        env_cpus = os.environ.get("CURTIN_VMTEST_NR_CPUS", None)
+        if env_cpus:
+            nr_cpus = env_cpus
+            logger.debug('Setting cpus from '
+                         ' env["CURTIN_VMTEST_NR_CPUS"] value: %s', nr_cpus)
+        if not nr_cpus:
+            nr_cpus = 1
+            logger.debug('Setting cpus to default value: %s', nr_cpus)
+
+        return str(nr_cpus)
+
+    @classmethod
     def setUpClass(cls):
         # check if we should skip due to host arch
         if cls.arch in cls.arch_skip:
@@ -535,7 +558,8 @@
             dowait = "--dowait"
 
         # create launch cmd
-        cmd = ["tools/launch", "--arch=" + cls.arch, "-v", dowait]
+        cmd = ["tools/launch", "--arch=" + cls.arch, "-v", dowait,
+               "--smp=" + cls.get_config_smp()]
         if not cls.interactive:
             cmd.extend(["--silent", "--power=off"])
 
@@ -646,6 +670,10 @@
                 fp.write(json.dumps({'grub': {'update_nvram': True}}))
             configs.append(grub_config)
 
+        if cls.dirty_disks and storage_config:
+            logger.debug("Injecting early_command to dirty storage devices")
+            configs.append(cls.dirty_disk_config)
+
         excfg = os.environ.get("CURTIN_VMTEST_EXTRA_CONFIG", False)
         if excfg:
             configs.append(excfg)
@@ -789,6 +817,7 @@
                target_disks + extra_disks + nvme_disks +
                ["--", "-drive",
                 "file=%s,if=virtio,media=cdrom" % cls.td.seed_disk,
+                "-smp",  cls.get_config_smp(),
                 "-m", "1024"])
 
         if not cls.interactive:

=== modified file 'tests/vmtests/test_basic.py'
--- tests/vmtests/test_basic.py	2017-04-26 16:14:04 +0000
+++ tests/vmtests/test_basic.py	2017-05-15 15:58:10 +0000
@@ -8,6 +8,8 @@
 
 class TestBasicAbs(VMBaseClass):
     interactive = False
+    nr_cpus = 2
+    dirty_disks = True
     conf_file = "examples/tests/basic.yaml"
     extra_disks = ['128G', '128G', '4G']
     nvme_disks = ['4G']

=== modified file 'tests/vmtests/test_bcache_basic.py'
--- tests/vmtests/test_bcache_basic.py	2017-04-26 16:14:04 +0000
+++ tests/vmtests/test_bcache_basic.py	2017-05-15 15:58:10 +0000
@@ -9,6 +9,8 @@
         "s390x",  # lp:1565029
     ]
     conf_file = "examples/tests/bcache_basic.yaml"
+    nr_cpus = 2
+    dirty_disks = True
     extra_disks = ['2G']
     collect_scripts = [textwrap.dedent("""
         cd OUTPUT_COLLECT_D

=== modified file 'tests/vmtests/test_mdadm_bcache.py'
--- tests/vmtests/test_mdadm_bcache.py	2017-05-10 21:29:20 +0000
+++ tests/vmtests/test_mdadm_bcache.py	2017-05-15 15:58:10 +0000
@@ -262,6 +262,8 @@
                      ('md1', 0)]
     active_mdadm = "2"
     uefi = True
+    nr_cpus = 2
+    dirty_disks = True
 
 
 class TrustyTestMirrorbootPartitionsUEFI(relbase.trusty,

=== modified file 'tools/launch'
--- tools/launch	2017-04-24 21:20:53 +0000
+++ tools/launch	2017-05-15 15:58:10 +0000
@@ -41,6 +41,7 @@
       -n | --netdev     netdev can be 'user' or a bridge
       -p | --publish F  make file 'F' available in web server
            --silent     use -nographic
+      -s | --smp S      use smp S for number of guest cpus (defaults to 2)
            --vnc D      use -vnc D (mutually exclusive with --silent)
                         directly through to qemu-system.
                         Note, qemu adds 5900 to port numbers. (:0 = port 5900)
@@ -349,8 +350,9 @@
 }
 
 main() {
-    local short_opts="a:A:d:h:i:k:n:p:v"
-    local long_opts="add:,append:,arch:,bios:,disk:,dowait,help,initrd:,kernel:,mem:,netdev:,no-dowait,power:,publish:,root-arg:,silent,serial-log:,uefi-nvram:,verbose,vnc:"
+    local short_opts="a:A:d:h:i:k:n:p:s:v"
+    local
+    long_opts="add:,append:,arch:,bios:,disk:,dowait,help,initrd:,kernel:,mem:,netdev:,no-dowait,power:,publish:,root-arg:,silent,serial-log:,smp:,uefi-nvram:,verbose,vnc:"
     local getopt_out=""
     getopt_out=$(getopt --name "${0##*/}" \
         --options "${short_opts}" --long "${long_opts}" -- "$@") &&
@@ -358,7 +360,7 @@
         { bad_Usage; return 1; }
 
     local seed=""
-    local bootimg="" bootimg_dist="" target="" mem="1024"
+    local bootimg="" bootimg_dist="" target="" mem="1024" smp="2"
     local udata="" ip="" http_port="${HTTP_PORT}" burl=""
     local tmp="" top_d
     local initrd="" kernel="" uappend="" iargs="" disk_args=""
@@ -412,6 +414,7 @@
                --root-arg) root_arg="$next";;
                --serial-log) serial_log="$next"; shift;;
                --silent) video="-nographic";;
+            -s|--smp) smp="$next"; shift;;
             --uefi-nvram|--bios)
                 # handle all --opt=* pass through here.
                 pt[${#pt[@]}]="$cur=$next";;
@@ -725,6 +728,7 @@
     # -monitor stdio
     cmd=(
         xkvm "${pt[@]}" "${netargs[@]}" --
+        -smp ${smp}
         -m ${mem} ${serial_args} ${video}
         -drive "file=$bootimg,if=none,cache=unsafe,format=qcow2,id=boot,index=0"
         -device "virtio-blk,drive=boot"