lp:~smoser/curtin/trunk-lp1731709-curthooks-write-files

Branch merges

Propose for merging

No branches dependent on this one.

Merged into lp:~raharper/curtin/trunk-lp1731709-curthooks-write-files at revision 545

Ryan Harper: Pending requested 2017-11-17

Diff: 240 lines (+43/-125)

6 files modified

curtin/commands/curthooks.py (+2/-20)
curtin/futil.py (+14/-0)
examples/tests/curthooks_writefiles.yaml (+0/-40)
tests/unittests/helpers.py (+13/-0)
tests/unittests/test_curthooks.py (+14/-29)
tests/vmtests/test_curthooks_write_files.py (+0/-36)

api

Related source package recipes

No recipes using this branch. (?)

Create packaging recipe

Related snap packages

Related bugs

Related blueprints

544. By Scott Moser on 2017-11-17

suggested changes

a.) put only the import into curthooks.write_files, not a definition.
    move the legacy definition to futil.
b.) drop the vmtests
c.) drop the mocks in the added unit tests, actually read the files
    that are written and verify they have expected content.
d.) raise a DeprecatedWarning if the function is called.

543. By Ryan Harper on 2017-11-14

Fix vmtest for curthooks.write_files()

542. By Ryan Harper on 2017-11-13

Re-add curthooks.write_files method for backwards compat

In revno 512, curthooks.write_files() was re-implemented under
curtin.futil.write_files and supports the original dictionary
format as well as other modes. This broke legacy images which used
curthooks that invoked curthooks.write_files. This branch restores
the function and behavior while utilizing the change in implementation.

Add unittests and vmtest to exercise the code path.

LP: #1731709

541. By Ryan Harper on 2017-11-07

Drop Precise from vmtest

It's dead Jim (Precise is EOL)

540. By Ryan Harper on 2017-11-02

clear_holders: bcache log IO/OS exceptions but do not raise

While stopping bcache devices, curtin races with the kernel w.r.t how
quickly it removes the bcache device (and the sysfs tree related).
The result is that as curtin attempts to write "1" into the sysfs tree
of bcache device to stop it, the range of errors changes depending on
when we race with the kernel asynchronously removing the sysfs path.
Therefore we log the exception errno we got but do not re-raise. The
the calling process is watching whether the same sysfs path is being
removed; if the bcache device fails to go away then curtin will have a
log of the exceptions to debug.

539. By Scott Moser on 2017-10-30

vmtest: Support newer qemu and multipath.

If you pass the same backing device for a disk to qemu, newer
versions (2.10+) will fail. In order to allow this, it requires
you to pass 'file.locking=off'.

This change makes the multipath tests add a 'file.locking=off' argument
to the disks that are actually intended to be added more than once.
xkvm is modified to remove that flag if the qemu being used does not
support it.

In order to accomplish this:
a.) make launch use xkvm '--disk' for all its disks rather than
    passing -drive and -device. This is good in that we know have
    a single way of passing disks around.
b.) fix a logic bug in xkvm when --disk= specified format= on it.

vmtest and launch now only use '--disk=' args, which are interpreted
by xkvm rather than using a mix of '--disk=' and '-drive'.

538. By Ryan Harper on 2017-10-30

block: enable control over exclusive_open use when wiping volumes

The exclusive_open check on a device is very useful to determine if
curtin is removing all metadata from a device w.r.t storage
configuraiton. However, in some cases during creation of new storage
configations when curtin attempts to add additional partitions to a
device that has partitions already added to an raid device, the
exclusive open is blocked while we attempt to zero some data on the
underlying device. The Mirrorboot-UEFI vmtest exemplifies this
situation.

This patch exposes a flag to the zero_file_at_offset method to allow
callers (block_meta in this case) to disable the exclusive_open. This
lets curtin wipe data at specific offsets on a volume without
requiring exclusive ownership of the device.

537. By Ryan Harper on 2017-10-26

block: handle wiping bcache parts

In some cases block devices may have bcache parts which are owned by the
block layer but are not assembled into a bcache device. The result is that
curtin fails to get exclusive opens when attempting to wipe the device.
Resolve this by testing if a volume has a 'bcache' directory under sysfs
and issuing the appropriate stop commands before wiping.

It's also possible for old "buried" metadata for bcache or lvm or raid
at particular partition offsets. Once curtin starts partitioning the
device the bcache kernel layer "finds" the parts and claims device
ownership and prevents wiping of the partition. We resolve this issue
by having curtin wipe 1M at the location of the partition before we
actually create it.

Fix 'curtin clear-holders' subcommand.

LP: #1718699

536. By Ryan Harper on 2017-10-23

vmtests: Defer ArtfulNetworkMtu SkipbyDate to 2018

Unlikely to expect an fix for LP: #1671951 before then.

535. By Ryan Harper on 2017-10-17

bcache: accept sysfs write failure in shutdown handler if path missing

Bcache device shutdown is asynchronus. The kernel may remove the
device after we've tested to see if the device needs to be stopped but
before we issue a 'stop' command to the bcache sysfs file.

When that happens, curtin throws an exception; this patch catches
those and tests if the target path is missing; and if so, ignores the
error as it's expected due to device removal.

All other write failures where the path is still present are still
valid and reported.

LP: #1700564