systemd-networkd needs to ensure DNS is up before network-online.target

The attachment "xenial_resolvconf-lp1649931.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

How is this fixed in the development release?

Also the having a "Regression Potential" of low just because not a lot of people use the service is wrong. A bad change could still horribly impact the users of the software, regardless of how many users there are.

On Thu, Dec 15, 2016 at 10:27 PM, Brian Murray <email address hidden> wrote:

> How is this fixed in the development release?
>

There is an upstream resolvconf bug that addresses that package portion:

 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=847440

> Also the having a "Regression Potential" of low just because not a lot
> of people use the service is wrong. A bad change could still horribly
> impact the users of the software, regardless of how many users there
> are.
>

I don't disagree (that saying few folks impacted means low).

This portion was split out from the previous systemd SRU

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636912

in which the changes were classified in the same way, so I was coping over
the assessment done by pitti and Foundations with which I agree.

I'd be happy if someone else could comment on whether they disagree with
the assessment.

IMO the risk is low:

The changes involve running resolvconf.service earlier (which sets an
inotify watch) and
does not regress anyone AFAICT.

Asking that DNS services (systemd-networkd-resolvconf-update.service) to
run before
network-online.target vs. (how it is today) running after
network-online.target is reach
 also has low regression potential since _nothing_ requiring networking
should
run before systemd reaches network-online.target anyhow.

These potential users would already be broken since the units don't
currently
ensure that resolvconf (when using networkd instead of
networking.service/ifupdown)
runs to completion before reaching network-online.target.

>
> ** Changed in: resolvconf (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1649931
>
> Title:
> systemd-networkd needs to ensure DNS is up before network-
> online.target
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/resolvconf/+
> bug/1649931/+subscriptions
>

On Thu, Dec 15, 2016 at 10:27 PM, Brian Murray <email address hidden> wrote:

> How is this fixed in the development release?
>

For systemd portion, we carry the
systemd-networkd-resolvconf-update.service as a delta from Debian

Ryan

On Thu, Dec 15, 2016 at 09:44:32PM -0000, Ryan Harper wrote:
> On Thu, Dec 15, 2016 at 10:27 PM, Brian Murray <email address hidden> wrote:

> > How is this fixed in the development release?

> For systemd portion, we carry the
> systemd-networkd-resolvconf-update.service as a delta from Debian

I'm not sure what you mean here; the systemd package in zesty is currently
in sync with Debian.

resolvconf (1.79ubuntu4) zesty; urgency=medium

  * debian/resolvconf.service: Add missing Wants=network-pre.target.

 -- Martin Pitt <email address hidden> Thu, 08 Dec 2016 10:21:12 +0100

resolvconf (1.79ubuntu3) zesty; urgency=medium

  * debian/resolvconf.service: resolvconf service must run before any of the
  possible network configuration services (ifupdown, networkd, NetworkManager)
  so use network-pre.target instead of service names. (LP: #1636912)

 -- Ryan Harper <email address hidden> Wed, 07 Dec 2016 03:30:55 -0600

On Thu, Dec 15, 2016 at 09:43:48PM -0000, Ryan Harper wrote:
> I don't disagree (that saying few folks impacted means low).

> This portion was split out from the previous systemd SRU

> https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636912

> in which the changes were classified in the same way, so I was coping over
> the assessment done by pitti and Foundations with which I agree.

> I'd be happy if someone else could comment on whether they disagree with
> the assessment.

> IMO the risk is low:

The point here is that 'regression potential' is not meant to be a scalar,
it's meant to be a description of where regressions are possible so that we
can assess whether additional testing is needed to guard against those
regressions.

And also to ensure that the SRU submitter has carefully thought about this
question, since going through that exercise usually results in a better SRU.

>> For systemd portion, we carry the
>> systemd-networkd-resolvconf-update.service as a delta from Debian

> I'm not sure what you mean here; the systemd package in zesty is currently
> in sync with Debian.

Ok, it appears that post-16.04, systemd-networkd-resolvconf-update.service is dropped from the package in favor of debian/extra/units/systemd-resolved.service.d/resolvconf.conf, so this patch does not apply.

I have verified that there is no ordering bug post-16.04, provided that the changes to resolvconf are applied. /lib/systemd/system/systemd-resolved.service.d/resolvconf.conf causes systemd-resolved to notify resolvconf of its nameserver, if /run/resolvconf/enable-updates exists. /run/resolvconf/enable-updates exists if resolvconf.service is started. resolvconf.service is started Before=network-pre.target; network.target is After=network-pre.target; systemd-resolved.service is After=network.target. So only the change to resolvconf is needed for yakkety and beyond.

Hello Ryan, or anyone else affected,

Accepted resolvconf into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resolvconf/1.78ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I accepted systemd into xenial-proposed but I received an LP timeout when the script was trying to add a comment. I image you can figure out what to do from the resolvconf comment though.

This has had systemd 231-9ubuntu3 stacked on top into xenial-proposed. See bug 1647485 for details. When verifying, please test and verify systemd 231-9ubuntu3, not the previous version, and confirm the package version you tested in your comment to avoid any misunderstanding. Thanks!

Something isn't working here. I also typo'd the test-case, in particular, the change to the cloud-init.service unit didn't specify 'systemd-networkd-wait-online.service', the sed command missed the 'd' in networkd.

When I fix that, I find that systemd-networkd-resolvconf-update.service does NOT run. Manually adding that to /lib/systemd/system/resolvconf.service.wants fixes the networkd boot path, but of course breaks the ifupdown path.

systemd-networkd-resolvconf-update.service uses default dependencies which will only run after sysinit.target, however cloud-init.service needs to run before sysinit.target; systemd apparently dropped running the service due to the inconsistent order. Adding a DefaultDependencies=no to systemd-networkd-resolvconf-update.{service,path} allows cloud-init.service to run early with DNS before sysinit.target.

I've marked verification failed for this bug since the current proposed packages do not resolv the issue without changes to systemd package (/lib/systemd/system/systemd-networkd-resolvconf-update.{service,path}.

Hello Ryan, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've verified the SRU test with the package in Xenial proposed. I've also updated an UbuntuCore16 image with uses networkd and cloud-init to validate that early networking which requires dns for resolving urls works.

This bug was fixed in the package resolvconf - 1.78ubuntu4

---------------
resolvconf (1.78ubuntu4) xenial; urgency=medium

  * debian/resolvconf.service: resolvconf service must run before any of the
    possible network configuration services (ifupdown, networkd,
    NetworkManager) so use network-pre.target instead of service names.
    (LP: #1649931)

 -- Ryan Harper <email address hidden> Wed, 07 Dec 2016 03:03:18 -0600

The verification of the Stable Release Update for resolvconf has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

For yakkety resolvconf update; I've updated the SRU test-case to include a more general netplan config for bringing up ethernet devices; the previous version only worked on lxd (or instances with devices named eth0).

With the above change; it looks like yakkety/zesty (due to running systemd-resolved instead of resolvconf service) need an update to the systemd-resolved service the instance is configured to use networkd insteadof ifupdown to ensure that systemd-resolved runs *before* network-online.target.

# diff -u systemd-resolved.service.orig systemd-resolv
--- systemd-resolved.service.orig 2017-03-06 19:37:44.206692449 +0000
+++ systemd-resolved.service 2017-03-06 19:33:09.467080158 +0000
@@ -12,6 +12,7 @@
 Documentation=http://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
 Documentation=http://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 After=systemd-networkd.service network.target
+Before=network-online.target

 # On kdbus systems we pull in the busname explicitly, because it
 # carries policy that allows the daemon to acquire its name.

This makes things more complicated for this SRU; yakkety/zesty will need an updated systemd; the resolvconf portion of this SRU (for yakkety) is still OK but it's not sufficient to achieve the goal of ensuring DNS is functional *before* we reach 'network-online.target'.

> I've verified the SRU test with the package in Xenial proposed.

Which package and what versions please? This shows up as green in https://people.canonical.com/~ubuntu-archive/pending-sru.html for systemd, but it's not clear to me from this bug that systemd 229-4ubuntu17 is ready to be released to xenial-updates.

Details about different release vs. network backend requirements

@racb

I verified systemd 229-4ubuntu17 with both ifupdown and networkd configuration.

The specific packages and features is somewhat complicated due to
the different scenarios and change in networking behavior in yakkety+

I've attached a chart with details. Also viewable here:

http://paste.ubuntu.com/24140363/

The tl;dr is for Xenial, we won't need another SRU as
systemd-resolved is not enabled to manage DNS, rather
resolvconf still controls /etc/resolv.conf

In Yakkety+, systemd-resolved ends up owning /etc/resolv.conf
and employs resolvconf to register itself therein.

I've moved the systemd-resolved issue for unit ordering out of this bug/SRU into a new one so we can finish out this SRU and handle this last change to systemd-resolved separately.

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1673860

Running the test case from the bug description for a yakkety container, this is the behavior I see with the yakkety-updates version of nplan:

-- Logs begin at Thu 2017-04-20 05:38:36 UTC, end at Thu 2017-04-20 05:40:08 UTC. --
Apr 20 05:38:36.302406 y1 systemd[1]: Started Nameserver information manager.
Apr 20 05:38:36.570955 y1 systemd[1]: resolvconf.service: Failed to reset devices.list: Operation not permitted
Apr 20 05:38:37.434155 y1 systemd[1]: systemd-networkd-wait-online.service: Failed to reset devices.list: Operation not permitted
Apr 20 05:38:37.434853 y1 systemd[1]: Starting Wait for Network to be Configured...
Apr 20 05:38:37.443689 y1 systemd-networkd-wait-online[218]: ignoring: lo
Apr 20 05:38:37.564242 y1 systemd[1]: systemd-networkd-wait-online.service: Failed to reset devices.list: Operation not permitted
Apr 20 05:38:37.564969 y1 systemd[1]: resolvconf.service: Failed to reset devices.list: Operation not permitted
Apr 20 05:38:50.119398 y1 systemd[1]: Started Wait for Network to be Configured.
Apr 20 05:38:50.120176 y1 systemd[1]: Reached target Network is Online.
Apr 20 05:38:50.132004 y1 systemd[1]: systemd-networkd-wait-online.service: Failed to reset devices.list: Operation not permitted

That doesn't appear to match the expected "bad" behavior. Ryan, do you know what I'm missing here?

I'm assuming it's a container permissions error, and give a VM a try.

I've tested this now in a 16.10 VM, and I can't verify this SRU because I can't confirm the buggy behavior. Adjusting the journalctl command for changed unit names in 16.10, I get:

# journalctl -o short-precise --unit resolvconf.service --unit network-online.target --unit systemd-networkd-wait-online.service --unit systemd-networkd.service --no-pager
-- Logs begin at Thu 2017-04-20 07:07:07 UTC, end at Thu 2017-04-20 07:08:09 UTC. --
Apr 20 07:07:08.973024 ubuntu systemd[1]: Starting Network Service...
Apr 20 07:07:09.013394 ubuntu systemd-networkd[958]: Enumeration completed
Apr 20 07:07:09.013744 ubuntu systemd[1]: Started Network Service.
Apr 20 07:07:09.015259 ubuntu systemd[1]: Starting Wait for Network to be Configured...
Apr 20 07:07:09.023517 ubuntu systemd-networkd[958]: ens3: IPv6 enabled for interface: Success
Apr 20 07:07:09.026605 ubuntu systemd-networkd[958]: ens3: Gained carrier
Apr 20 07:07:09.030066 ubuntu systemd-networkd[958]: ens3: DHCPv4 address 192.168.122.138/24 via 192.168.122.1
Apr 20 07:07:09.030329 ubuntu systemd-networkd[958]: ens3: Not connected to system bus, ignoring transient hostname.
Apr 20 07:07:10.100496 ubuntu systemd-networkd[958]: ens3: Gained IPv6LL
Apr 20 07:07:22.841001 ubuntu systemd-networkd[958]: ens3: Configured
Apr 20 07:07:22.843432 ubuntu systemd-networkd-wait-online[971]: ignoring: lo
Apr 20 07:07:22.849021 ubuntu systemd[1]: Started Wait for Network to be Configured.
Apr 20 07:07:22.850384 ubuntu systemd[1]: Reached target Network is Online.

Notably, I don't see any output here about resolvconf.

# systemctl status resolvconf.service --no-pager
● resolvconf.service - Nameserver information manager
   Loaded: loaded (/lib/systemd/system/resolvconf.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2017-04-20 07:14:42 UTC; 4min 25s ago
     Docs: man:resolvconf(8)
 Main PID: 659 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/resolvconf.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

I don't know why the journal is being rotated.

Ok, I've succeeded in verifying this by adding an artificial ExecStartPre=/bin/sleep 10 to both old and new resolvconf.service. Old:

# journalctl -o short-precise --unit resolvconf.service --unit network-online.target --unit systemd-networkd-wait-online.service --unit systemd-networkd.service --no-pager
-- Logs begin at Thu 2017-04-20 07:36:15 UTC, end at Thu 2017-04-20 07:36:37 UTC. --
Apr 20 07:36:17.265818 ubuntu systemd[1]: Starting Network Service...
Apr 20 07:36:17.430595 ubuntu systemd-networkd[905]: Enumeration completed
Apr 20 07:36:17.437376 ubuntu systemd[1]: Started Network Service.
Apr 20 07:36:17.440060 ubuntu systemd[1]: Starting Wait for Network to be Configured...
Apr 20 07:36:17.460259 ubuntu systemd-networkd[905]: ens3: IPv6 enabled for interface: Success
Apr 20 07:36:17.473412 ubuntu systemd-networkd[905]: ens3: Gained carrier
Apr 20 07:36:17.499888 ubuntu systemd-networkd[905]: ens3: DHCPv4 address 192.168.122.138/24 via 192.168.122.1
Apr 20 07:36:18.934247 ubuntu systemd-networkd[905]: ens3: Gained IPv6LL
Apr 20 07:36:25.978756 ubuntu systemd[1]: Started Nameserver information manager.
Apr 20 07:36:31.522527 ubuntu systemd-networkd[905]: ens3: Configured
Apr 20 07:36:31.531655 ubuntu systemd[1]: Started Wait for Network to be Configured.
Apr 20 07:36:31.533566 ubuntu systemd[1]: Reached target Network is Online.

New:
# journalctl -o short-precise --unit resolvconf.service --unit network-online.target --unit systemd-networkd-wait-online.service --unit systemd-networkd.service --no-pager
-- Logs begin at Thu 2017-04-20 07:38:31 UTC, end at Thu 2017-04-20 07:39:01 UTC. --
Apr 20 07:38:41.872259 ubuntu systemd[1]: Started Nameserver information manager.
Apr 20 07:38:41.890403 ubuntu systemd[1]: Starting Network Service...
Apr 20 07:38:41.930968 ubuntu systemd-networkd[987]: Enumeration completed
Apr 20 07:38:41.931769 ubuntu systemd[1]: Started Network Service.
Apr 20 07:38:41.934063 ubuntu systemd[1]: Starting Wait for Network to be Configured...
Apr 20 07:38:41.944754 ubuntu systemd-networkd[987]: ens3: IPv6 enabled for interface: Success
Apr 20 07:38:41.950236 ubuntu systemd-networkd[987]: ens3: Gained carrier
Apr 20 07:38:41.957193 ubuntu systemd-networkd[987]: ens3: DHCPv4 address 192.168.122.138/24 via 192.168.122.1
Apr 20 07:38:43.444868 ubuntu systemd-networkd[987]: ens3: Gained IPv6LL
Apr 20 07:38:56.057060 ubuntu systemd-networkd[987]: ens3: Configured
Apr 20 07:38:56.060665 ubuntu systemd-networkd-wait-online[1010]: ignoring: lo
Apr 20 07:38:56.069850 ubuntu systemd[1]: Started Wait for Network to be Configured.
Apr 20 07:38:56.076092 ubuntu systemd[1]: Reached target Network is Online.

In neither case did I end up with an unconfigured /etc/resolv.conf , but this does at least demonstrate the capacity for the race (and its resolution). Marking verified.

This bug was fixed in the package resolvconf - 1.79ubuntu1.1

---------------
resolvconf (1.79ubuntu1.1) yakkety; urgency=medium

  * debian/resolvconf.service: resolvconf service must run before any of the
    possible network configuration services (ifupdown, networkd,
    NetworkManager) so use network-pre.target instead of service names.
    (LP: #1649931)

 -- Ryan Harper <email address hidden> Wed, 07 Dec 2016 03:30:55 -0600

This bug was fixed in the package systemd - 229-4ubuntu17

---------------
systemd (229-4ubuntu17) xenial; urgency=medium

  * Make systemd-networkd-resolvconf-update.{service,path} run earlier
    to ensure DNS is configured prior to reaching network-online.target.
    (LP: #1649931)

 -- Ryan Harper <email address hidden> Thu, 02 Feb 2017 15:41:48 -0600