lp:~powersj/curtin/make-tox-great-again

Branch merges

Propose for merging

No branches dependent on this one.

Merged into lp:~curtin-dev/curtin/trunk at revision 454

Server Team CI bot: Approve (continuous-integration) on 2017-02-07

curtin developers: Pending requested 2017-02-07

Diff: 39 lines (+12/-2)

2 files modified

Makefile (+4/-2)
tox.ini (+8/-0)

api

Related source package recipes

No recipes using this branch. (?)

Create packaging recipe

Related snap packages

Related bugs

Related blueprints

451. By Joshua Powers on 2017-02-07

Remove style checking during buil and add latest style checks to tox

- make check will no longer run the style checks
- created style-check in makefile target to continue to run pep8 and
pyflakes
- added tox envs 'tip-pycodestyle' and 'tip-pyflakes' to run latest
style checking

LP: #1652329

450. By Ryan Harper on 2017-02-07

helpers/common: Add grub install debugging output

At various times (bug 1642298) we have needed to see more information
on what is being done during grub installation.
Adding execution tracing exposes the exact command that's run for
debugging.

Also, add a dump of efibootmgr output before and after to aide
in debugging UEFI grub installs.

Capture this output and send it to log at debug level rather than
letting it go to standard output / error.

449. By Scott Moser on 2017-02-07

flake8: remove unused variable.

448. By Ryan Harper on 2017-02-06

curtin: add version module and display in output and logs

It's difficult to determine what curtin version is running
from the collected logs in a maas deployment which uses
curtin by sending it to the target system. Resolve this by
having curtin announce itself.

- Introduce a new version module which does the following:
  - allows packaging to set the version by source code modification.
    packaging changed to replace a string @@PACKAGED_VERSION@@ with
    the version packaged.
  - if no packaged version is set, and .bzr dir exists, then
    run bzr revno to get information on that.
  - if no bzr, then use the static version.
- Modify curtin.command.main to print the version string in argparse output
- Update curtin logging setup to emit a version string as soon as logging
  is configured

447. By Christian Ehrhardt  on 2017-02-06

vmtest: Add the ability to add extra config files to test execution.

This is what I used to test multipath-tools via curtin, but it can be much
more helpful by generally allowing to add any external content

446. By Ryan Harper on 2017-02-06

content decoding in load_file, apply_net raise exception on errors

This patch series fixes two issues. First, subcommands of the apply_net
command were exiting non-zero but we failed to raise and exception
which let curtin hide an error. We've modified apply_net to re-raise
the exception when it occurs. Additionally update vmtest to look for
stack-traces in the installation log and mark a test failed if it
detects one; this should prevent future cases from re-occurring.

The second error is when loading a file with and encoding, load_file
did not handle this case. Merge in a version of load_file from
cloud-init which already handles this case. Introduce new
unittests to validate the function as it touches some block related
code where we read partition data directly.

Finally, the mock_open feature of unittest.mock only supports
binary data in version 2.0.0 or newer, so skip this unittest on
systems without a new-enough mock; note this does not affect
the function of the code on the same release, only the unittest.

445. By Ryan Harper on 2017-02-03

gpg: retry when recv'ing gpg keys fail

Retry gpg recv operations in case error is transient.
Add unittests to validate curtin.gpg including retry behavior

444. By Ryan Harper on 2017-02-01

Add clear_holders checks to disk and partition handlers

In some cases disks may contain block device metadata outside of the
current partition table. Due to Linux udev rules and behaviors;
creating or modifying a disks partition table may trigger udev rules
which may start up mdadm, bcache or lvm services which will obtain
exclusive access to the disk preventing installation from completing.

This branch addresses this issue with the following

- block.wipe_volume now uses os.O_EXCL flag to obtain exclusive access
  to a device; this fails when something else (like mdadm) obtains
  a file handle on the device with O_EXCL flag

- Upon failure to obtain exclusive access, we dump information about
  a devices block holders

- During disk handling, after creating a partition table we check for
  holders and then clear them if present.

- During partition handling, after creating a new partition we check
  for holders and then clear them if present.

443. By Ryan Harper on 2017-01-27

vmtest: overhaul image sync

Changes to support HWE and Centos images have exposed issues in the image
sync process. Our 'make sync-images' was using a simplestreams filter
that did not pickup any of the required HWE kernels which meant that
during vmtest runs we would trigger new image syncs. Compounding this
issue was the fact that the filter used was too-wide which picked up
things like di-initrd,di-kernel files.

Additionally for an empty IMAGE_DIR repository, there were two bugs; one
it assumed that it could load the vmtest.json/vmtest-centos.json files.
Second when the repo was empty, we did not return an iterable type.

The following changes have been made to achieve the goals:
  - make sync-images should download everything we need to run a complete
    vmtest run without acquiring any new files from our image server.
  - Handle empty repositories properly
  - Do not attempt to sync with the streams on each test, but only if
    specific needed files are missing
  - Do not sync streams during jenkings-runner; the previous
    make sync-images should have pulled everything

442. By Ryan Harper on 2017-01-26

vmtest: skip apt-proxy test if not set

Testing of apt-proxy requires one to be set in the host. If a proxy
is not set then the test-case will fail. This was discovered on diglett
host which usally has an apt-proxy but recently it was disabled which
tripped this failure.