Merge ~raharper/cloud-init:fix/lp-1766287-run-after-udev-settle-service into cloud-init:master

[Ryan Harper]

cloud-init-local: use systemd-udev-settle.service for stable nic names

The cloud-init-local.service expects that any network device name changes
have already been completed by the kernel or udev daemon. While at least
in Ubuntu the systemd-udev-settle.service is enabled, nothing in the
Ubuntu cloud images by default has a Wants for this service so it does not
run.

In some situations we've found that the renaming of interfaces from kernel
names (eth0, eth1, etc) to their persistent names (eno1, ens3, enp0s1,
etc) may happen after cloud-init-local has started where it reads values
from sysfs about what network devices are present, and which device to use
as a fallback nic.

Subsequently, cloud-init-local would write out network configuration for a
kernel device name which would no longer be present by the time that
networking services start to bring up the devices. The result is that the
instance does not get networking configured. Prior to use of
systemd-networkd, the Ubuntu 'networking.service' unit included a call to
udevadm settle which is why this race is not seen on a Xenial system.

LP: #1766287

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:9bbca03312b23839d67f039d817351cc59b21364
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1055/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1055/rebuild

review: Approve (continuous-integration)

[Dimitri John Ledkov (xnox)]

I don't like this, as this will introduce a boot delay and total time to first boot for everybody, even if not strickly needed.

E.g. cloud-init may have networking config disabled (like it is on some agent managed clouds); clouds that don't do network things in cloud-init-local do not neeed this. Nor one needs this if the networking devices are initialized by now.

There are APIs such that one can query, if the network interfaces names are all there, and if they have been processed by udev and hence are renamed into a final name / ready.

One could connect to netlink/udev and check that things are finalized. And at that point one could have an asyncio loop to await for the states to be finalized. Or in a more functional style, call a blocking trigger settle on the net subsystem. But only do this at the point were one reached the decision point to start doing stuff based on the interface names and realized that they are nonfinal.

Also, if the names maybe getting renamed, the ifindexes will not. So a lot of early userspace code that does things to networks does so by ifindex instead, if that is sufficient for the things done there.

ps. I am assuming that there are datasources for which correct and valid network config is generated, based on datasource expectations irrespective if the names are ready or not. If there is no common-case datasource that does this, things may be different.

[Dimitri John Ledkov (xnox)]

"and <increase> total time to first..."
strictly

[Ryan Harper (raharper)]

In the case where networking is disabled in cloud-init and the cloud has their own; the race still exists; that is if the network names don't match the configuration in the image when the networking is applied; it will fail.

There are no APIs that I'm aware of that tell you if an interface has been processed by udev;
The kernel will report the name_assign_type = 4 if userspace has applied a rename;

The blocking trigger is interesting, however, the race in this case is that an existing net subsystem trigger has already run (systemd-udev-trigger.service); so it seems wrong to trigger the net subsystem *again* after it's already applying any rules and the uevents are in-flight.

While it's interesting that ifindex don't change, none of our network configuration formats (eni, sysconfig, netplan) accept config by ifindex, so that won't help us.

We certainly don't want to add time to boot just because; I think there is a strong argument that we want cloud-init-local to run at a time when any udev rules which could rename interfaces has been applied. I'm open to alternatives, but systemd-udev-settle.service certainly seems very useful way to ensure this state.

[Scott Moser (smoser)]

network config by ifindex would be insane. it potentially changes every boot.

[Dimitri John Ledkov (xnox)]

I am glad this was superseded by https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/344339

review: Needs Fixing

Unmerged commits

9bbca03... by Ryan Harper on 2018-04-24

cloud-init-local: use systemd-udev-settle.service to ensure stable nic names

The cloud-init-local.service expects that any network device name changes have
already been completed by the kernel or udev daemon. While at least in Ubuntu
the systemd-udev-settle.service is enabled, nothing in the Ubuntu cloud images
by default has a Wants for this service so it does not run.

In some situations we've found that the renaming of interfaces from kernel names
(eth0, eth1, etc) to their persistent names (eno1, ens3, enp0s1, etc) may happen
after cloud-init-local has started where it reads values from sysfs about what
network devices are present, and which device to use as a fallback nic.

Subsequently, cloud-init-local would write out network configuration for a
kernel device name which would no longer be present by the time that networking
services start to bring up the devices. The result is that the instance does
not get networking configured. Prior to use of systemd-networkd, the Ubuntu
'networking.service' unit included a call to udevadm settle which is why this
race is not seen on a Xenial system.

LP: #1766287

diff --git a/systemd/cloud-init-local.service.tmpl b/systemd/cloud-init-local.service.tmpl
index ff9c644..2babf05 100644
--- a/systemd/cloud-init-local.service.tmpl
+++ b/systemd/cloud-init-local.service.tmpl
@@ -3,6 +3,8 @@
 Description=Initial cloud-init job (pre-networking)
 {% if variant in ["ubuntu", "unknown", "debian"] %}
 DefaultDependencies=no
+Wants=systemd-udev-settle.service
+After=systemd-udev-settle.service
 {% endif %}
 Wants=network-pre.target
 After=systemd-remount-fs.service