~raharper/curtin:feature/vmtests-v3streams

Branch merges

Propose for merging

Merged into curtin:master at revision 92e45a8bd98444bf92d15dc4781599dabb1ca8a7

Scott Moser (community): Approve on 2018-01-11

Server Team CI bot: Approve (continuous-integration) on 2018-01-11

Diff: 1960 lines (+551/-193)

48 files modified

examples/tests/allindata.yaml (+4/-4)
examples/tests/basic.yaml (+4/-4)
examples/tests/basic_iscsi.yaml (+1/-1)
examples/tests/bcache-wipe-xfs.yaml (+4/-4)
examples/tests/bcache_basic.yaml (+2/-2)
examples/tests/dirty_disks_config.yaml (+10/-0)
examples/tests/lvm.yaml (+1/-1)
examples/tests/lvm_iscsi.yaml (+1/-1)
examples/tests/mdadm_bcache.yaml (+3/-3)
examples/tests/mdadm_bcache_complex.yaml (+3/-3)
examples/tests/mdadm_iscsi.yaml (+1/-1)
examples/tests/mirrorboot-msdos-partition.yaml (+2/-2)
examples/tests/mirrorboot-uefi.yaml (+2/-2)
examples/tests/mirrorboot.yaml (+2/-2)
examples/tests/nvme.yaml (+1/-1)
examples/tests/nvme_bcache.yaml (+2/-2)
examples/tests/raid10boot.yaml (+4/-4)
examples/tests/raid5bcache.yaml (+5/-5)
examples/tests/raid5boot.yaml (+3/-3)
examples/tests/raid6boot.yaml (+4/-4)
examples/tests/uefi_basic.yaml (+3/-3)
tests/vmtests/__init__.py (+150/-23)
tests/vmtests/image_sync.py (+6/-5)
tests/vmtests/releases.py (+24/-0)
tests/vmtests/test_apt_config_cmd.py (+1/-1)
tests/vmtests/test_apt_source.py (+1/-1)
tests/vmtests/test_basic.py (+20/-4)
tests/vmtests/test_bcache_basic.py (+10/-2)
tests/vmtests/test_centos_basic.py (+18/-1)
tests/vmtests/test_install_umount.py (+1/-1)
tests/vmtests/test_iscsi.py (+10/-2)
tests/vmtests/test_journald_reporter.py (+2/-17)
tests/vmtests/test_lvm.py (+10/-2)
tests/vmtests/test_lvm_iscsi.py (+12/-0)
tests/vmtests/test_lvm_root.py (+1/-1)
tests/vmtests/test_mdadm_bcache.py (+79/-11)
tests/vmtests/test_mdadm_iscsi.py (+9/-1)
tests/vmtests/test_multipath.py (+10/-2)
tests/vmtests/test_network.py (+6/-3)
tests/vmtests/test_network_vlan.py (+3/-3)
tests/vmtests/test_nvme.py (+20/-4)
tests/vmtests/test_old_apt_features.py (+1/-1)
tests/vmtests/test_raid5_bcache.py (+10/-2)
tests/vmtests/test_simple.py (+9/-2)
tests/vmtests/test_ubuntu_core.py (+1/-1)
tests/vmtests/test_uefi_basic.py (+11/-3)
tools/jenkins-runner (+7/-3)
tools/launch (+57/-45)

api

Related source package recipes

No recipes using this branch. (?)

Create packaging recipe

Related snap packages

Related charm recipes

92e45a8... by Ryan Harper on 2017-12-19

vmtests: switch to MAAS v3 streams for images and kernels

Switching to v3 streams enables testing of Xenial HWE and HWE-edge
kernels and the squashfs root which MAAS 2.3 uses.

- Add v3 streams URL
- Introduce two attributes, ephemeral_ftype, target_ftype to determine
  which files are needed for the ephemeral environment and which for the
  target
- Add kflavor and subarch filters to streams query and release classes to
  support kernel variants: ga, hwe, hwe-edge. We're currently only testing
  subarch 'generic' and ignoring 'low-latency'
  - When testing with hwe and hwe-edge, emit a kernel configuration to
    curtin to ensure we install the matching kernel in the target image
  - Make sure vmtests append collect scripts to the base class parameter
    so all tests run the BaseVM collect_scripts (dpkg-query)
- Introduce new release classes, XenialGA, XenialHWE and XenialEdge
  Note, XenialGA replaces the general Xenial release class, and that
  not all test-cases run Xenial HWE/Edge kernels; most of the storage
  ones do.
- Booting from a squashfs image requires some changes
  - publish the squashfs image and boot VMs with root=URL (except trusty)
  - utilize overlayfs to keep the ephemeral environment pristine and
    use the squashfs image as a the curtin install source.
  - Add some boot parameters to the ephemeral environment to enable
    networking, including DNS for package installation.
  - Use BOOTIF= parameter for multi-nic vmtests to ensure we only bring
    up one nic for ephemeral networking. Sometimes multiple nics would
    come online and then cause networking hiccups which prevented
    installs from completing.
- Modify vmtest storage configurations for disks to all use a 'serial'
  attribute. Currently we use 'disk-X' as the serial value where X
  increments from 'a' -> 'z' as the test needs.
- jenkins-runner now prints additional information about what images
  are found and available.
- tools/launch
  - modify --append to allow multiple parameters to grow the array.
  - fix bug in --publish which prevented more than a single file being
    published
  - allow PUBURL to be expanded in root-arg
  - add a 'boot-image' parameter which is exported to the VM with a serial
    number to provide a consistent path to the device,
    /dev/disk/by-id/virtio-boot-disk
  - drop '2G' boot size parameter to boot disks, instead allow qemu-img to
    default to the size of backing image.

be9ab3c... by Scott Moser on 2018-01-09

tests: update pack tests to clear out pyc files more completely.

When the test test_curtin_help_has_hacked_version was first written,
the file .pyc file that was known to be created for
my_path/my.py was my_path/my.pyc

Newer versions of python (at least in cpython 3.6), create:
   version/__pycache__/version.cpython-36.pyc

This updates the test to remote those pyc files also.

The failure to remove the .pyc file results in a race condition
where sometimes the .pyc file is seen as valid even though the
.py file was written after it.

c5cb03e... by Scott Moser on 2017-12-19

debian/control: drop conflicts that had bzr version

this is fine as the bzr version (~bzr54) is not in any supported
release.. trusty is at 126-0ubuntu1 in released version even.

8108efb... by Scott Moser on 2017-12-19

Switch uses of bzr to git, borrow from cloud-init git workflow.

Basically replace all references to bzr. git-describe and experience
in cloud-init make this fairly painless.

 - curtin/version.py
   replace usage of 'bzr revno' with 'git-describe'.
   versions 'packed' from git will show a version like 17.1-5-gdfd3389f.

 - debian/changelog.trunk
   remove the hard coded version string.

 - tools/export-tarball, tools/make-tarball:
   replace export-tarball with a version of make-tarball taken from
   cloud-init.

 - tools/build-deb, update to use git-describe and make-tarball.
   Also set some default parameters (-S -d -us -uc).

 - doc/, tox.ini, just replace bzr with git.

 - tools/new-upstream-snapshot: copy from cloud-init debian/

6fa5028... by Scott Moser on 2017-12-15

Release 17.1

bzr-revno: 553

bbfff08... by Scott Moser on 2017-12-08

packaging: update debian/build-deb to remove ~bzrREV when using equal tag

When building a curtin package from tree, if the current revno matches
a tag revision, use the tag value and drop the ~bzrREV from the version
 string.

bzr-revno: 552

0361d29... by Scott Moser on 2017-12-08

pack: fix packing when curtin is installed inside a snap.

When curtin is installed as part of a MAAS snap, then it would fail
to find its helpers directory. That is because instead of helpers
living in usr/lib/curtin/helpers they will be in
 /snap/maas/<version>/usr/lib/curtin/helpers
But helpfully, SNAP will be set in the environment to point to
 /snap/maas/<version>/

The fix is to adjust pack to trust the environment variable
and pack up the installed files based on that prefix.

bzr-revno: 551

5486375... by Scott Moser on 2017-12-08

tox: move to pylint 1.7.4

Update the pylint version used in tox from 1.5.4 to 1.7.4.
The primary gain from this is the ability to run with python3.6.
  https://github.com/PyCQA/pylint/issues/1113

bzr-revno: 550

aa6f250... by Scott Moser on 2017-12-07

merge with trunk

bzr-revno: 548.1.3

0f3ec13... by Scott Moser on 2017-12-07

update per ryan feedback

bzr-revno: 548.1.2