transient systemd ordering cycle in boot with overlayroot

open-iscsi test utilizes overlayroot to boot a cloud-image with root
filesystem on a read-only iscsi server.

The /etc/fstab file in the image looks like this:
  LABEL=cloudimg-rootfs / ext4 defaults 0 0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

when init takes over from the initramfs, we have

  /proc/cmdline:
    nomodeset iscsi_initiator=maas-enlist
    iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
    iscsi_target_port=3260 iscsi_initiator=maas-enlist
    ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
    root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
    overlayroot=tmpfs console=ttyS0
    ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

  /etc/fstab:
    # cat /etc/fstab
    #
    # This fstab is in an overlay. The real one can be found at
    # /media/root-ro/etc/fstab
    # The original entry for '/' and other mounts have been updated to be placed
    # under /media/root-ro.
    # To permanently modify this (or any other file), you should change-root into
    # a writable view of the underlying filesystem using:
    # sudo overlayroot-chroot
    #
    #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
    /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
    #LABEL=UEFI /boot/efi vfat defaults 0 0

 /root/root-ro/etc/fstab:
    LABEL=cloudimg-rootfs / ext4 defaults 0 0
    #LABEL=UEFI /boot/efi vfat defaults 0 0

 /proc/mounts:
   sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
   proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
   udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
   devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
   tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
   /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0
   tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
   overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

 /proc/1/mountinfo:
   21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
   22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
   23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755
   24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
   25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755
   26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
   27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
   28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following:

 mkdir /media/root-ro /media/root-rw # in the initramfs
 mount -t tmpfs tmpfs-root /media/root-rw
 mkdir /media/root-rw/overlay-workdir/_
 mount --move $ROOTMNT /media/root-ro
 mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root
 mkdir $ROOTMNT/media/root-ro
 mkdir $ROOTMNT/media/root-rw
 mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
 mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

 # then, if 'ro' on the command line, it mounts /root read-only.
 mount -o remount,ro $ROOTMNT

The script then exits, as ROOTMNT is now set up with a read-only mount
of the overlayroot. All the mounts it has done have been moved
under ROOTMNT.

On failure systemd reports:
   [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start
   [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
   [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start
   [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided

On successful boot, we can login and see:

$ find /run/systemd/ -name "*.mount" | xargs ls -l
-rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/generator/-.mount
lrwxrwxrwx 1 root root 10 Aug 21 13:55 /run/systemd/generator/local-fs.target.requires/-.mount -> ../-.mount
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-hugepages.mount -> b847fc2d06b54b1ab86a6f526d149522
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-mqueue.mount -> f991d3bfe35d4e2bb9aa86677ff31d70
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-fs-fuse-connections.mount -> a561df591a28437f990f94269fa08be9
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-config.mount -> 141e9bbaca9041e48458749e29bc58fb
lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-debug.mount -> a406163f265c4b02b37e6f92ddbfc029

$ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done
== /run/systemd/generator/-.mount ==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
Where=/
What=/media/root-ro/
Type=overlay
Options=lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_

--
[1] https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot

Related bugs:
 * bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: systemd 237-3ubuntu10
ProcVersionSignature: Ubuntu 4.17.0-6.7-generic 4.17.9
Uname: Linux 4.17.0-6-generic x86_64
ApportVersion: 2.20.10-0ubuntu7
Architecture: amd64
Date: Tue Aug 21 14:06:24 2018
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
ProcEnviron:
 TERM=vt220
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.11.1-1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-cosmic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-cosmic:cvnQEMU:ct1:cvrpc-i440fx-cosmic:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-cosmic
dmi.sys.vendor: QEMU