lp:~raharper/curtin

Author: Ryan Harper
Author Date: 2018-08-17 14:37:58 UTC

Adjust comments around vmtest config creation

Author: Ryan Harper
Author Date: 2018-08-10 21:57:08 UTC

Skip install of known broken scenario.

Author: Ryan Harper
Author Date: 2018-08-06 16:31:49 UTC

xkvm: add virtio-based rng devices reading from /dev/urandom

Author: Ryan Harper
Author Date: 2018-07-23 19:14:23 UTC

Drop exception variable, unused

Author: Ryan Harper
Author Date: 2018-07-18 22:01:00 UTC

Add retry to clear_holders.assert_clear() and unittest to reproduce issue

Author: Ryan Harper
Author Date: 2018-06-15 21:27:03 UTC

vmtests: network_mtu move fixby date out 4 months from last value

The upstream changes needed to handle mixed ipv4/ipv6 mtu have not
been completed in networkd so this test is currently failing.
Move our skipby date out 4 months.

Author: Ryan Harper
Author Date: 2018-06-15 21:26:18 UTC

Drop unused determine_partition_number and reduce vmtests for partition order

Author: Ryan Harper
Author Date: 2018-06-03 23:54:41 UTC

Fix tip-pyflakes imported but unused call to util.get_platform_arch

Fix tox -e tip-pyflakes complaint about unused import.

Author: Ryan Harper
Author Date: 2018-05-18 19:12:08 UTC

releasing curtin version 18.1-17-gae48e86f-0ubuntu1~16.04.1

Author: Ryan Harper
Author Date: 2018-05-18 19:02:29 UTC

releasing curtin version 18.1-17-gae48e86f-0ubuntu1~17.10.1

Author: Ryan Harper
Author Date: 2018-05-18 19:00:24 UTC

releasing curtin version 18.1-17-gae48e86f-0ubuntu1~18.04.1

Author: Ryan Harper
Author Date: 2018-05-18 17:43:56 UTC

Replace use of .assert_called() with checking .call_count

Author: Ryan Harper
Author Date: 2018-05-17 18:06:33 UTC

Add unittests for zfs.zfs_supported and update mock for clear-holders use of zfs_supported

Author: Ryan Harper
Author Date: 2018-05-17 01:03:43 UTC

Fix remaining clear-holder issues with rescanning partitions and iscsi devices

Moving the scan to run before issuing the next shutdown command satisfies the
LVM scenario where msdos extended partitiosn disappear and further we need to
always trigger the automatic mode for iscsi devices as the iscsiadm discovery
reconfigures the defaults.

Author: Ryan Harper
Author Date: 2018-05-09 22:49:25 UTC

Add Bionic ZfsRootFs variant

Author: Ryan Harper
Author Date: 2018-05-11 20:28:06 UTC

Add missing trailing )

Author: Ryan Harper
Author Date: 2018-04-30 06:29:48 UTC

util.get_efibootmgr: filter bootorder by found entries

In some cases a previous EFI configuration may reference boot
items that are no longer present which results in curtin
invoking efibootmgr with invalid parameters:

% efibootmgr -o 0001,0000,3000,2001,2002,2003
Invalid BootOrder order entry value0001,0000,3000,2001,2002,2003

Curtin will update the the order list by dropping any elements
that don't have an actual boot entry.

LP: #1767584

Author: Ryan Harper
Author Date: 2018-04-22 22:16:32 UTC

Add debug logging to test_dname base class

Author: Ryan Harper
Author Date: 2018-01-29 21:41:58 UTC

make_dname for bcache should use backing device uuid

bcache minor numbers are not stable across reboots which means
the dname symlinks pointing to a bcacheN isn't going to be stable either.
Instead, build a udev rule that links the backing device's bcache uuid
to the assigned dname. Additionally also update the backing device
'label' sysfs attribute with the dname value.

This does rely on changes to bcache-tools to trigger the rules, currently
there is a kernel bug (LP: #1729145) which is pending, and a user-space
workaround in bcache-tools (LP: #1728742). Either is sufficent for
the updated rules to be effective.

Some additional fixes found during this work

command/curthooks.py
  - don't append .rule to rule files, they already have .rule
examples/tests/mdadm_bcache_complex.yaml
  - removed, duplicate of mdadm_bcache.yaml
tests/vmtests/
  - refactor test_dname into base class, fix subclasses

Author: Ryan Harper
Author Date: 2018-04-17 20:54:30 UTC

s,byid,by-id in zpool documentation

Author: Ryan Harper
Author Date: 2018-04-18 15:15:57 UTC

releasing curtin version 18.1-5-g572ae5d6-0ubuntu1

Author: Ryan Harper
Author Date: 2018-04-17 23:29:51 UTC

Add extra disk for test

Author: Ryan Harper
Author Date: 2018-04-16 18:36:46 UTC

Add unittests to validate install exception kept when unmount:disabled set

Author: Ryan Harper
Author Date: 2018-04-16 16:31:35 UTC

Refactor release check to be forward looking

Author: Ryan Harper
Author Date: 2018-03-29 14:25:47 UTC

releasing curtin version 18.1-1-g45564eef-0ubuntu1~16.04.1

Author: Ryan Harper
Author Date: 2018-03-29 14:16:12 UTC

releasing curtin version 18.1-1-g45564eef-0ubuntu1~17.10.1

Author: Ryan Harper
Author Date: 2018-03-23 15:52:01 UTC

releasing package curtin version 17.1-51-gbd40234f-0ubuntu1

Author: Ryan Harper
Author Date: 2018-03-09 08:12:45 UTC

vmtest: skip some bcache tests pending kernel fixes

There are a few kernel bugs related to bcache which prevent testing of
some bcache scenarios, add skip_by_date for bug 1728742 and bug 1754581.

Author: Ryan Harper
Author Date: 2018-03-20 22:46:56 UTC

helpers/shell-archive: drop use of tar --sparse flag

Do not use tar [-S, --sparse] flag: at least one filesystem (btrfs)
requires sync/fsync to ensure proper detection of sparse files.
curtin doesn't contain any sparse files so this does not affect payload
size.

LP: #1757565

Author: Ryan Harper
Author Date: 2018-03-21 00:24:00 UTC

vmtests: extend skip_by_date for lvmroot_xfsroot and network_mtu

Reported bugs are not yet fixed, move out by 3 months, post
18.04 for possible fix.

Author: Ryan Harper
Author Date: 2018-03-07 14:44:24 UTC

zfs/zpool version=28 by default, allow users to override default values.

Include documentation note and link to openzfs wiki for furthe details
around compatibility and features.

LP: #1754013

Author: Ryan Harper
Author Date: 2018-03-19 16:37:52 UTC

vmtest: fix pollinate re.search to use groupdict()

In python3.6 re.search() returns an SRE_Match which has
support for dictionary indexing, however on older python3
we do not. Fallback to the supported SRE_Match.groupdict()
before indexing named matches.

Author: Ryan Harper
Author Date: 2018-02-07 20:23:39 UTC

Add pollinate user-agent configuration support

The pollinate client in Ubuntu images supports custom data to be
appended to the user-agent value sent when contacting the pollen
server. This patch allows users to add any key/value pairs which
will be added to the pollinate client configuration.

By default, curtin will configure pollinate with the curtin version
and if being deployed by MAAS, the MAAS version. Users can also
disable configuration of the pollinate client.

Additional fixes
vmtest:
  report_webhook_logger didn't encode writes when handling GET request

Author: Ryan Harper
Author Date: 2018-03-05 08:02:54 UTC

Disable dirty_disks mode for mdadm + iscsi tests

Two issues with dirty_disk mode. First, after the early
stage runs, we need to tear down the raid array so we can
disconnect the iscsi devices, second even when that is done
there is a bug in mdadm version in Artful and Bionic which
prevents successful testing (LP: #1753786)

Author: Ryan Harper
Author Date: 2018-02-23 14:41:03 UTC

clear_holders: wipe complex devices before disassembly

The curtin clear-holders code did not wipe the contents of the
assembled devices prior to shutting them down. In cases where
curtin re-assembles the same devices, in a RAID for example, then
when the RAID device comes on-line, it may prevent some tools from
creating new metadata on the target device. In particular, the bcache tools
do not want to overwrite an existing bcache superblock resulting in
a failed deployment when layering a bcache over a RAID volume.

The fix is in two parts. First, clear-holders will wipe the superblock of
the assembled device before it breaks it apart. Second, when creating a
bcache backing device, check first if the target device is already claimed
by the bcache layer; if so then raise a RuntimeError as clear holders should
have removed this device.

curtin.block
 - expose 'exclusive' boolean and pass it through to context manager

curtin.block.clear_holders
 - Add logic to wipe assembled devices before calling shutdown function
 - introduce internal _wipe_superblock which only contains the retry
   logic
 - switch to using dmsetup remove for lvm destruction
 - improve identify_raid, it incorrectly identified partitions on RAID0 and
   RAID1 devices as raid devices, but they are actually partitions.

LP: #1750519

Author: Ryan Harper
Author Date: 2018-02-27 15:23:53 UTC

vmtests: network_mtu: fix bug number and move date out one month

The bug number on the network_mtu skip_by_date for missing IPV6 MTU support
in netplan/networkd was off-by-one; fix that and move date out one more month.

Author: Ryan Harper
Author Date: 2018-02-23 15:04:02 UTC

fix /usr/bin/curtin exit code in install failure

When curtin encounters an error during the 'install' stage, the default
configuration enables collecting of system logs. The collect_logs function
'create_log_tarfall' included a sys.exit(0) which should only be part of the
'collect_logs_main' method; this resulted in failed installs exiting with a
return code of 0. The fix is to move the sys.exit(0) up to the main method.

LP: #1751176

Author: Ryan Harper
Author Date: 2018-02-13 13:05:49 UTC

docs: Document use of format command with fstype: zfsroot

Add documentation for use of 'zfsroot' in the format command.
Note that zfsroot support is experimental. Include an example
zfsroot configuration.

Author: Ryan Harper
Author Date: 2018-02-13 12:32:18 UTC

vmtest: fix centos root-tgz image sync

Remove all remaining uses of vmtest.root-tgz as part of our local mirror.
This fixes syncing of centos images which use ftype: root-tgz.

Author: Ryan Harper
Author Date: 2018-01-17 15:49:31 UTC

clear-holders: detect and remove devices from kernel swap as needed

During testing of subiquity the ephemeral environment may activate
existing swap partitions which then prevent curtin from using a device during
storage configuration. This branch introduces a change to TestBasic to add
a swap partition and a command to the dirty-disk mode to forcibly enable
the swap device to reproduce the error. The fix involves having
clear_holders detect and remove the device (disk or partition) from
the kernel swap via use of swapoff.

LP: #1743643

Author: Ryan Harper
Author Date: 2018-01-26 20:49:42 UTC

Add zfsroot as a filesystem type

Enable simpler use of zfsroot by tagging the root filesystem format with
'zfsroot'. Curtin will then automatically construct zpool and zfs commands
to create a ZFS on Root setup. Some validation of the config included:
 * only one 'zfsroot' type is allowed.
 * zfsroot entry must be referenced by the / mount.

Author: Ryan Harper
Author Date: 2018-01-29 18:59:27 UTC

zfs: only load zfs on xenial+ skip artful/bionic zfs env export

The zfs module is only available on xenial+, so guard against modprobing it on
precise and trusty.

Xenial is the only release which curtin is exporting an environment variable to
aid grub to find zfs devices, therefor we should only check if the environment
has the exported value on xenial, skiptest otherwise.

Author: Ryan Harper
Author Date: 2017-12-20 21:27:45 UTC

Add zpool, zfs storage commands for experimental support of ZFS on Root

curtin/block
 - Add get_dev_disk_byid() to return a mapping of devname to disk/by-id paths
 - Add zfs and zpool to install deps dictionary

curtin/block/clear_holders
 - Add modprobe zfs

curtin/block/zfs
 - implement zpool_create, zfs_create, zfs_list, zfs_export and zfs_mount
   commands

curtin/dep
 - Add zfs, zfsutils-linux to ephemeral environment as needed
 - Introduce a check for required kernel modules (zfs)

curtin/commands/block_meta
 - implement handlers for type: zpool and zfs
 - add get_poolname resolver

curtin/commands/curthooks
 - For xenial, use apt-mark hold zfs-dkms in target to prevent installation
 - Add an injection of a zfs environment variable required for ZFS on rootfs
   to work with grub; allows grub to extract the path to the zfs vdevs full
   path rather than just the devname (/dev/disk/by-id/foo vs disk/by-id/foo)

curtin/commands/install
 - Export ZPOOL_VDEV_NAME_PATH=1 into target environment to allow grub to
   work with zfs on root

doc/topics/storage.rst
 - Add documentation for zpool, zfs commands and experimental ZFS-on-Root

helpers/common
 - Update install_grub to skip block-device check if target mountpoint is zfs.

tests/unittests/test_block
 - Add test for disk_byid methods

tests/unittests/test_block_zfs
 - Add coverage for block.zfs

tests/unittests/test_clear_holders
 - Update test to account for modprobe zfs

tests/vmtests/test_zfsroot
 - Add initial zfsroot install and test

Author: Ryan Harper
Author Date: 2018-01-18 21:48:09 UTC

releasing package curtin version 17.1-11-ga4c9636b-0ubuntu1~17.10.1

Author: Ryan Harper
Author Date: 2018-01-18 21:30:16 UTC

releasing package curtin version 17.1-11-ga4c9636b-0ubuntu1~16.04.1

Author: Ryan Harper
Author Date: 2018-01-18 17:37:32 UTC

vmtest: Drop Zesty release

Zesty is no longer supported as of 2018-01-13[1]

1. https://lists.ubuntu.com/archives/ubuntu-announce/2018-January/000227.html

Author: Ryan Harper
Author Date: 2018-01-11 20:41:10 UTC

vmtest: initialize logger with class names for easy parsing

When running vmtest in parallel, having each message include the
TestClass name helps developers map the output to specific tests when
reading the combined log.

Author: Ryan Harper
Author Date: 2018-01-16 21:42:48 UTC

releasing package curtin version 17.1-6-g8b145067-0ubuntu1

Author: Ryan Harper
Author Date: 2018-01-16 21:34:12 UTC

releasing package curtin version 17.1-6-g8b145067-0ubuntu1~17.10.1

Author: Ryan Harper
Author Date: 2018-01-16 21:24:50 UTC

releasing package curtin version 17.1-6-g8b145067-0ubuntu1~16.04.1

Author: Ryan Harper
Author Date: 2018-01-08 22:03:54 UTC

mirrorboot validation

Author: Ryan Harper
Author Date: 2017-12-19 21:50:09 UTC

vmtests: switch to MAAS v3 streams for images and kernels

Switching to v3 streams enables testing of Xenial HWE and HWE-edge
kernels and the squashfs root which MAAS 2.3 uses.

- Add v3 streams URL
- Introduce two attributes, ephemeral_ftype, target_ftype to determine
  which files are needed for the ephemeral environment and which for the
  target
- Add kflavor and subarch filters to streams query and release classes to
  support kernel variants: ga, hwe, hwe-edge. We're currently only testing
  subarch 'generic' and ignoring 'low-latency'
  - When testing with hwe and hwe-edge, emit a kernel configuration to
    curtin to ensure we install the matching kernel in the target image
  - Make sure vmtests append collect scripts to the base class parameter
    so all tests run the BaseVM collect_scripts (dpkg-query)
- Introduce new release classes, XenialGA, XenialHWE and XenialEdge
  Note, XenialGA replaces the general Xenial release class, and that
  not all test-cases run Xenial HWE/Edge kernels; most of the storage
  ones do.
- Booting from a squashfs image requires some changes
  - publish the squashfs image and boot VMs with root=URL (except trusty)
  - utilize overlayfs to keep the ephemeral environment pristine and
    use the squashfs image as a the curtin install source.
  - Add some boot parameters to the ephemeral environment to enable
    networking, including DNS for package installation.
  - Use BOOTIF= parameter for multi-nic vmtests to ensure we only bring
    up one nic for ephemeral networking. Sometimes multiple nics would
    come online and then cause networking hiccups which prevented
    installs from completing.
- Modify vmtest storage configurations for disks to all use a 'serial'
  attribute. Currently we use 'disk-X' as the serial value where X
  increments from 'a' -> 'z' as the test needs.
- jenkins-runner now prints additional information about what images
  are found and available.
- tools/launch
  - modify --append to allow multiple parameters to grow the array.
  - fix bug in --publish which prevented more than a single file being
    published
  - allow PUBURL to be expanded in root-arg
  - add a 'boot-image' parameter which is exported to the VM with a serial
    number to provide a consistent path to the device,
    /dev/disk/by-id/virtio-boot-disk
  - drop '2G' boot size parameter to boot disks, instead allow qemu-img to
    default to the size of backing image.

Author: Ryan Harper
Author Date: 2017-12-20 01:00:25 UTC

Fix tools/build-deb

- Import some tools from cloud-init
  - tools/read-version
    - Updated to mention curtin instead
    - Added --shell which emits key=value pairs
    - Added --long which emits just the version_long value
  - tools/make-tarball
    - Updated to mention/use curtin instead
- Removed tools/export-tarball
- debian/changelog.trunk
  - Modified for easier replacement for the version string
- Removed all of the bzr command and revision parsing logic
  and now use read-version to extract the long and short
  version values and feed them to make-tarball

This now works and writes out debs like:

 curtin_17.1-2-g757848bf-1~bddeb_all.deb
 curtin-common_17.1-2-g757848bf-1~bddeb_all.deb
 python3-curtin_17.1-2-g757848bf-1~bddeb_all.deb
 python-curtin_17.1-2-g757848bf-1~bddeb_all.deb