[2.3] resolv.conf is not set (during commissioning or testing)

Can you include /proc/cmdline used?

cloud-init is configuring networking from the kernel cmdline, as ip= is seen on the cmdline

2017-08-18 22:56:13,931 - stages.py[INFO]: Applying network configuration from cmdline

ubuntu@withkvm:~$ cat /proc/cmdline
BOOT_IMAGE=ubuntu/amd64/generic/xenial/daily/boot-kernel nomodeset root=squash:http://192.168.122.2:5248/images/ubuntu/amd64/generic/xenial/daily/squashfs ro ip=::::withkvm:BOOTIF ip6=off overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:{'datasource_list': ['MAAS']}end_cc cloud-config-url=http://192.168.122.2:5240/MAAS/metadata/latest/by-id/xh33gw/?op=get_preseed apparmor=0 log_host=192.168.122.2 log_port=514 initrd=ubuntu/amd64/generic/xenial/daily/boot-initrd BOOTIF=01-52-54-00-03-02-6e

The cloud-init output shows that networking was up and working (shows a gateway and hitting the ubuntu archive). I wonder if the initial DHCP lease that happened in the initramfs was up and it dropped the lease?

There are no timestamps shown in your log, but guessing by the time stamp of your paste, you ran the pastebinit command ~ 13 minutes after cloud-init had finished.

Scott,

What information are you looking with the time stamps so I can reproduce
this and get you that exact info?

On Mon, Aug 21, 2017 at 4:31 PM Scott Moser <email address hidden> wrote:

> There are no timestamps shown in your log, but guessing by the time
> stamp of your paste, you ran the pastebinit command ~ 13 minutes after
> cloud-init had finished.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1711760
>
> Title:
> [2.3] resolv.conf is not set (during commissioning)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1711760/+subscriptions
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Well, can you just collect as much as you can..

Immediately after the system is up, ssh in. dns should work (as it most certainly did work for cloud-init running 'apt-get update'.).

So ssh in, and collect
  journalctl -o short-precise
  /var/log/cloud-init*
  /var/lib/dhcp/
  dmesg

if you can get console... i'd love that.

Basically, dns *was* functional, but then later it became non-functional. we need to figure out why.

This is easily reproducible by booting into any MAAS ephemeral environment(commissioning, testing, rescue mode). Apt works regardless of /etc/resolv.conf being set as apt is configured to proxy through the rack controller. MAAS is giving the DNS server over DHCP as running dhclient sets /etc/resolv.conf.

I've attached the files you requested, there is nothing in /var/lib/dhcp

On Fri, Aug 25, 2017 at 5:47 PM, Lee Trager <email address hidden>
wrote:

> This is easily reproducible by booting into any MAAS ephemeral
> environment(commissioning, testing, rescue mode). Apt works regardless
> of /etc/resolv.conf being set as apt is configured to proxy through the
> rack controller. MAAS is giving the DNS server over DHCP as running
> dhclient sets /etc/resolv.conf.
>
> I've attached the files you requested, there is nothing in /var/lib/dhcp
>

Can you tar up /run ? The initramfs is doing the initial network
configuration.

>
> ** Attachment added: "log.tar.xz"
> https://bugs.launchpad.net/maas/+bug/1711760/+attachment/
> 4938963/+files/log.tar.xz
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1711760
>
> Title:
> [2.3] resolv.conf is not set (during commissioning or testing)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1711760/+subscriptions
>

Attached is /run. One thing I forgot to mention. MAAS starts an ephemeral environment by passing a user_data script to cloud-init. This script downloads and runs all of the commissioning and testing scripts. My understand of cloud-init is that this means cloud-init still hasn't finished running as its running the script from MAAS.

The dyn-conf generator in the initramfs does this

## This file is generated by cloud-initramfs-dyn-netconf
auto lo
iface lo inet loopback
manual ens3
iface ens3 inet dhcp
dns-nameservers 10.0.0.2 0.0.0.0
dns-search maas

However, the resolvconf dir in /run/resolvconf does not have antything
either.

And while the proxy works for apt, it's clear during early boot that DNS is
not working:

/usr/lib/snapd/snapd[942]: snapmgr.go:411: Cannot prepare auto-refresh
change: Post https://search.apps.ubuntu.com/api/v1/snaps/metadata: dial
tcp: lookup search.apps.ubuntu.com on [::1]:53:

as snapd pokes URLs outside the system.

Reading through run dir, we have all of the settings, and it appears that
resolvconf does not get updated:

% cat resolvconf/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by
resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

This appears to be similar to this older bug:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1432829

Which relied on a systemd ifupdown helper to call resolvconf on the
interface that was brought up in initramfs.
I don't see that script as part of systemd anymore.

Also this.
https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1501033

Really feels like a regression in either the open-iscsi package, or systemd
for Xenial.

Ideally when this is fixed again, maas can add this as integration test to
catch if this breaks again.

On Mon, Aug 28, 2017 at 3:39 PM, Lee Trager <email address hidden>
wrote:

> Attached is /run. One thing I forgot to mention. MAAS starts an
> ephemeral environment by passing a user_data script to cloud-init. This
> script downloads and runs all of the commissioning and testing scripts.
> My understand of cloud-init is that this means cloud-init still hasn't
> finished running as its running the script from MAAS.
>
> ** Attachment added: "run.tar.xz"
> https://bugs.launchpad.net/maas/+bug/1711760/+attachment/
> 4940470/+files/run.tar.xz
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1711760
>
> Title:
> [2.3] resolv.conf is not set (during commissioning or testing)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1711760/+subscriptions
>

Ryan,
fwiw, I wrote at https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1713537/comments/1 a description of how all this is tied together in xenial->zesty. the ifupdown helper is still present, it has just moved.

Your /run does not have a /run/initramfs/open-iscsi.interface file
that file should have been written by the initramfs local-top code.
if it is not present, then resolvconf will not be run by
net-interface-handler [1] and /etc/resolv.conf not updated.

I'm not sure how that could have happened though.
the code that does this is at [2]

[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/net-interface-handler?h=applied/ubuntu/xenial-devel
[2] https://git.launchpad.net/~usd-import-team/ubuntu/+source/open-iscsi/tree/debian/extra/initramfs.local-top?h=applied/ubuntu/xenial-devel#n181

Scott,

Since we are no longer using iscsi and using the squashfs over HTTP, would open-iscsi even be used?

Possibly not then, the device that was in use has a net-<dev>.conf file;
 If we don't have iscsi then what other script is going to create the
.interface file?

Here's the conf file for ens3 (the device in use)

% cat run/net-ens3.conf
DEVICE='ens3'
PROTO='dhcp'
IPV4ADDR='10.0.0.54'
IPV4BROADCAST='10.0.0.255'
IPV4NETMASK='255.255.255.0'
IPV4GATEWAY='10.0.0.1'
IPV4DNS0='10.0.0.2'
IPV4DNS1='0.0.0.0'
HOSTNAME=''
DNSDOMAIN='maas'
NISDOMAIN=''
ROOTSERVER='10.0.0.2'
ROOTPATH=''
filename='pxelinux.0'
UPTIME='10'
DHCPLEASETIME='600'
DOMAINSEARCH=''

On Tue, Aug 29, 2017 at 10:19 AM, Blake Rouse <email address hidden>
wrote:

> Scott,
>
> Since we are no longer using iscsi and using the squashfs over HTTP,
> would open-iscsi even be used?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1711760
>
> Title:
> [2.3] resolv.conf is not set (during commissioning or testing)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1711760/+subscriptions
>

Blake.
that makes sense. I had forgotten about that switch to using rooturl.

So here is what happens. I do not think this is regression, though. I'm
not sure how it could have ever worked.

a.) initramfs (rooturl) configures networking due to 'ip=' on the command line.
this process writes files /run/network-eth0.conf with content shown in your
attached /run here.

b.) root pivots to new root (with the sqaushfs mounted loop), and keeps
/run intact.

c.) cloud-init reads the files in /run/network-eth0.conf and uses that
to override the fallback networking. In order to avoid bouncing the
interfaces configured in initramfs, cloud-init writes 'control-manual'
stanzas in ENI rather than 'auto eth0'. In the iscsi path, if cloud-init
wrote 'auto', then the nic might get bounced and iscsi root broken.

In the iscsi root case, its the 'net-interface-handler' that handles
transitioning the dns settings in /run/network-eth0.conf over to
the "real root". But in this case, nothing does that.

marked this 'confirmed' since I am pretty sure I understand the error. I'm not necessarily sure on the right way to fix it.

Just for future reference, it was unclear in my mind why the logs showed apt-get update succeeding which seemed to require dns (archive.ubuntu.com). The reason that worked was that http_proxy was set for apt. When using an http proxy, the client system does not resolve the Host, but rather leaves that to happen on the proxy.

Hi All,

So what's the latest status on this and/or what is the proposed fix?

Hi Andres --

This fix is complicated in that the networking involved with passing rooturl= and ip= gets into hairy bits in the distro and the handover from initramfs to the rootfs, not just cloud-init. It still needs to be discussed. We'll make sure we put an update on here once we have a more clear implementation plan.

This script runs and demonstrates functional dns.

It launches a guest with a custom kernel command line
pointing to a nocloud datasource running on the host.
It uses user mode networking to seed a dns server and a
dns search.

The 'seedfrom' is done over dns which would not work if this
change were put in place. To get that to work, you have to add
an entry in /etc/hosts for 10.0.2.2 (suggest 'qemu-host').

There is a PPA available at
  https://launchpad.net/~smoser/+archive/ubuntu/lp1711760
with the fix in place.

I've also asked the MAAS team to build a maas-images with an updated squashfs.

I made a couple changes to lp:maas-images on Friday to make that easy to do
 http://bazaar.launchpad.net/~maas-images-maintainers/maas-images/maas-ephemerals/revision/381
 http://bazaar.launchpad.net/~maas-images-maintainers/maas-images/maas-ephemerals/revision/382

So, to build a image and stream with this content, you essentially need to just set M2E_ADD_REPOS in the environment and then build as you would.

example:
  export M2E_ADD_REPOS=ppa:smoser/lp1711760

I have verified that smoser changes do work with MAAS.

Testing steps:

1. Generated xenial amd64 image stream using smosers PPA.
2. Installed MAAS.
3. Removed the workaround in MAAS.
4. Imported the xenial amd64 image with smosers PPA.
5. Entered rescue mode.
6. Pinged google.com. (Worked)
7. Exited rescue mode.
8. Ran internet connectivity test. (Passed)
9. Ran commissioning with SSH.
10. Pinged google.com. (Worked)

I think Andres has access to the GMAAS, would appreciate if he could
verify the fix there. If not, let me know and I can try it.

Thanks!

@Mark,

I do have access (just tested). I'll give it a try tomorrow! Thanks!

I updated some state on the tasks. As justification, this fix is in resolvconf, so I dropped cloud-init. I had added it thinking it might need integration, but it did not. Also marked the 'ubuntu' task as "won't fix". The problem did not currently exist in Artful, and the solution provided in resolvconf is not relevant there as resolvconf is not used in Artful.

Bumping this to critical provided that at this point, this completely breaks MAAS.

Also, I've tested this in GMAAS and confirm the fix works!

I have reviewed this. +1 from ~ubuntu-sru.

Regression risk is mitigated as Scott says by careful gating of the active code to limit exposure of these changes to the non-default use case only.

I did note the following:

What happens when the DHCP lease acquired in the initramfs expires? What if the DNS server changes after a new lease? Really it's the entire DHCP lease that needs handing over from the initramfs rather than just the DNS server that was provided at the time of the first lease on boot. But this will only affect this impacted non-default use case. If this doesn't matter for you, then fine. If it does, then please mark verification-failed and we can go again with a fix that will cover this case.

If not on systemd and inside a container, check will not work. We don't support !systemd on Ubuntu, so that should be OK. We may be on upstart after upgrading from Trusty; in that case the /proc/cmdline gate should still work so this will only affect systems booted this way using MAAS on Trusty and then upgraded to Xenial until next reboot.

Adding a udev rule that is a shell script for every network up and down. How does this effect udev, for example on a system with a large number of flapping interfaces? On a quick glance I didn't see any other shell script udev rule on such a wide reaching trigger.

Any race conditions before is-from-initrd gate, for example if multiple interfaces come up close together in time? I don't see any.

Please make sure to test that the gating is working correctly in the non-kernel-cmdline case during SRU verification.

Hello Andres, or anyone else affected,

Accepted resolvconf into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resolvconf/1.78ubuntu5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

> I have reviewed this. +1 from ~ubuntu-sru.

Thanks for the thoughtful review.

> What happens when the DHCP lease acquired in the initramfs expires? What
> if the DNS server changes after a new lease? Really it's the entire DHCP
> lease that needs handing over from the initramfs rather than just the
> DNS server that was provided at the time of the first lease on boot. But
> this will only affect this impacted non-default use case. If this
> doesn't matter for you, then fine. If it does, then please mark
> verification-failed and we can go again with a fix that will cover this
> case.

Agreed. This is not covered at this point, but never has been.
In the open-iscsi path, we've been using this code for the maas use case
since 12.04. Your point is valid, and we're hoping in 18.04 to correctly
transition this lease over the "real root".

The answer on "What happens" right now is "Nothing". The system will
continue to use its initramfs-provided dhcp address. There is no
persistent process that would re-evaluate the lease.

> If not on systemd and inside a container, check will not work. We don't
> support !systemd on Ubuntu, so that should be OK. We may be on upstart
> after upgrading from Trusty; in that case the /proc/cmdline gate should
> still work so this will only affect systems booted this way using MAAS
> on Trusty and then upgraded to Xenial until next reboot.

It doesn't specifically have to use systemd as init, just have the
systemd-detect-virt. You're right, we could have done something else
there, to cover a system that did not have that.

> Adding a udev rule that is a shell script for every network up and down.
> How does this effect udev, for example on a system with a large number
> of flapping interfaces? On a quick glance I didn't see any other shell
> script udev rule on such a wide reaching trigger.

Open-iscsi *does* have such a rule (the implementation is borrowed from
there). Also, open-iscsi is installed by default in all 16.04 images,
so while not being as ubiquitious as resolvconf, it is very common.

> Please make sure to test that the gating is working correctly in the
> non-kernel-cmdline case during SRU verification.

ACK.

the attached log shows this is working when expected and not when not expected.

I've verified these two manually! Thanks for all the work!

This bug was fixed in the package resolvconf - 1.78ubuntu5

---------------
resolvconf (1.78ubuntu5) xenial; urgency=medium

  * support reading dns information written by initramfs. (LP: #1711760)

 -- Scott Moser <email address hidden> Mon, 23 Oct 2017 12:44:53 -0400

The verification of the Stable Release Update for resolvconf has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

I've uploaded the same change that is in xenial but slightly adjusted for trusty.
I'm attaching an updated recreate script that works for both xenial and trusty.
(trusty needs to install the cloud-initramfs-dyn-netconf package to mimic the MAAS ephemeral environment).

Hello Andres, or anyone else affected,

Accepted resolvconf into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resolvconf/1.69ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

testing showed that I uploaded a bad version.
Fixed in a new upload
https://launchpadlibrarian.net/345007871/resolvconf_1.69ubuntu1.3_source.changes

Hello Andres, or anyone else affected,

Accepted resolvconf into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resolvconf/1.69ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package resolvconf - 1.69ubuntu1.3

---------------
resolvconf (1.69ubuntu1.3) trusty-proposed; urgency=medium

  * Fix bad shell syntax in newly added
    /lib/resolvconf/net-interface-handler (LP: #1711760)

resolvconf (1.69ubuntu1.2) trusty-proposed; urgency=medium

  * support reading dns information written by initramfs. (LP: #1711760)

 -- Scott Moser <email address hidden> Wed, 08 Nov 2017 15:47:00 -0500

Hello Andres, or anyone else affected,

Accepted resolvconf into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resolvconf/1.79ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've verified zesty with the script that is attached.
Output is shown here.

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

This bug was fixed in the package resolvconf - 1.79ubuntu4.1

---------------
resolvconf (1.79ubuntu4.1) zesty-proposed; urgency=medium

  * support reading dns information written by initramfs. (LP: #1711760)

 -- Scott Moser <email address hidden> Fri, 08 Dec 2017 14:47:54 -0500