Merge ~dbungert/curtin:lp-2057661-flock-zfs-ctd into curtin:master

[Dan Bungert]

do not squash

zfs: add flock for keystore

In LP: #2057661 and the (currently) 4 duplicates, the luksFormat of
keystore can randomly fail. stracing of udevd looking for
https://systemd.io/BLOCK_DEVICE_LOCKING/ shows that indeed udevd is
examining these keystore devices.

Lock the keystore, and add some `udevadm settle` to be more like
Ubiquity behavior.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:3fc197239e3a0ebb366e188c0d2a7c8e0a36bbaf
https://jenkins.canonical.com/server-team/job/curtin-ci/265/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-amd64/265/
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-arm64/265/
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-ppc64el/265/
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-s390x/265/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/curtin-ci/265//rebuild

review: Approve (continuous-integration)

[Michael Hudson-Doyle (mwhudson)]

I think this is fine and probably helps. I think it's a bit overbroad probably but maybe that's not a real problem considering how much overbroad stuff there is in curtin currently...

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:e4c7d7186c59ade359652db5accb6cb7156df499
https://jenkins.canonical.com/server-team/job/curtin-ci/266/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-amd64/266/
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-arm64/266/
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-ppc64el/266/
    SUCCESS: https://jenkins.canonical.com/server-team/job/curtin-ci/nodes=metal-s390x/266/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/curtin-ci/266//rebuild

review: Approve (continuous-integration)

diff --git a/curtin/block/zfs.py b/curtin/block/zfs.py
index ccdddbb..519fa51 100644
--- a/curtin/block/zfs.py
+++ b/curtin/block/zfs.py
@@ -12,6 +12,7 @@ from contextlib import ExitStack
 from pathlib import Path
 
 from curtin.config import merge_config
+from curtin.udev import udevadm_settle
 from curtin import distro
 from curtin import util
 from . import blkid, get_supported_filesystems
@@ -89,25 +90,34 @@ class ZPoolEncryption:
         zfs_create(
             self.poolname, "keystore", {"encryption": "off"}, keystore_size,
         )
+        keystore_volume = f"/dev/zvol/{self.poolname}/keystore"
+        udevadm_settle(exists=keystore_volume)
 
         with ExitStack() as es:
             for vdev in self.vdevs:
                 es.enter_context(util.FlockEx(vdev))
 
             # cryptsetup format and open this keystore
-            keystore_volume = f"/dev/zvol/{self.poolname}/keystore"
             cmd = ["cryptsetup", "luksFormat", keystore_volume, self.keyfile]
-            util.subp(cmd)
+
+            # strace has shown that udevd does indeed probe this keystore
+            with util.FlockEx(keystore_volume):
+                util.subp(cmd, capture=True)
+
+            udevadm_settle()
+
             dm_name = f"keystore-{self.poolname}"
             cmd = [
                 "cryptsetup", "open", "--type", "luks", keystore_volume,
                 dm_name, "--key-file", self.keyfile,
             ]
-            util.subp(cmd)
+            util.subp(cmd, capture=True)
+
+            dmpath = f"/dev/mapper/{dm_name}"
+            udevadm_settle(exists=dmpath)
 
         with ExitStack() as es:
             # format as ext4, mount it, move the previously-generated systemkey
-            dmpath = f"/dev/mapper/{dm_name}"
             es.enter_context(util.FlockEx(dmpath))
             cmd = ["mke2fs", "-t", "ext4", dmpath, "-L", dm_name]
             util.subp(cmd, capture=True)
diff --git a/curtin/util.py b/curtin/util.py
index 66735e8..d9b7962 100644
--- a/curtin/util.py
+++ b/curtin/util.py
@@ -1452,6 +1452,7 @@ class FlockEx:
             raise TimeoutError("Failed to acquire LOCK_EX on {self.device}")
 
     def __exit__(self, *args):
+        LOG.debug(f"Releasing fcntl LOCK_EX on {self.device}")
         with suppress(Exception):
             fcntl.flock(self.lock_fd, fcntl.LOCK_UN)
         with suppress(Exception):