Add trailing dot to make connectivity-check.ubuntu.com. absolute and reduce NXDOMAIN warning noise

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package network-manager - 1.24.2-1ubuntu2

---------------
network-manager (1.24.2-1ubuntu2) groovy; urgency=medium

  * debian/20-connectivity-ubuntu.conf: Add trailing dot to make
    connectivity-check.ubuntu.com. absolute and reduce NXDOMAIN
    warning noise, thanks MarchH (lp: #1880258)

 -- Sebastien Bacher <email address hidden> Mon, 29 Jun 2020 16:20:07 +0200

@ Seb

systemd behaviour could be better for domains without '.'
however, that is considered low, in the light of network-manager fix.

Imho network-manager fix should be SRUed, do you agree? And it would be on the desktop team to do so, right?

> systemd behaviour could be better for domains without '.'

One thing that would help with these confusing NXDOMAIN errors in general (not just from NetworkManager) is for systemd to provide a more verbose and possibly less "private"[*] mode that shows the requested domains somewhere without having to use tcpdump on the side and try to match timestamps, process IDs and what not. In fact this mode could be considered "more secure" because of the added admin transparency. It could be off by default for privacy reasons - depending on where the information goes.

Maybe this more verbose mode exists and I missed it? Or a feature request already filed? In either case please share, thanks.

[*] just guessing the current rationale

We discussed it in our meeting, it probably makes sense to include it in the next SRU we do but we don't believe it deserves rls tracking, updating the tag according to that decision

> One thing that would help with these confusing NXDOMAIN errors in general

the vast, VAST majority of the time you see this message it is actually not a NXDOMAIN error. This is due to a Ubuntu-only patch to systemd to work around some select captive portals that are slightly broken, so in any environment outside the broken captive portals (e.g. public wifi that you have to 'click here to accept terms' before getting internet access - and note that not all captive portals are broken) if you see this NXDOMAIN "error" it is almost always just a normal lookup of a domain that doesn't exist, and the error message is simply wrong (this also slows down dns due to forcing fallback to a lower dns protocol level and retry of the already-failed lookup).

To clarify specifically for this bug, the lookup of "connectivity-check.ubuntu.com.your_domain" clearly has nothing to do with any "DNS violation", and the NXDOMAIN returned by the upstream nameserver is the *correct* response - that hostname really, actually doesn't exist.

network-manager could work around this problematic Ubuntu-only systemd patch, but the real problem is unquestionably that systemd should not have the Ubuntu-only patch that's causing these messages.

Fixing this appropriately (i.e. so that systemd still works with the broken captive portal issue) requires access to one of the broken captive portals, so I haven't been able to work on correctly fixing this lately, but it is something I want to do, so we can get rid of the very unfortunate false NXDOMAIN "error" messages.

> the vast, VAST majority of the time you see this message it is actually not a NXDOMAIN error.

I don't have any extensive data as you seem to have but _for me at home_ (= office now!) and without zero captive portal involved, adding this trailing dot reduces the NXDOMAIN frequency from one message every 5 minutes to a few messages _per day_ (NM's default interval is 300 seconds)

The "journalctl --boot -p notice" total also becomes about twice smaller. I filed this bug for a reason.

> the NXDOMAIN returned by the upstream nameserver is the *correct* response - that hostname connectivity-check.ubuntu.com.your_domain really, actually doesn't exist.

Yes of course, hence this one character fix which IMHO also removes one large "distraction" from the main, captive portal issue!

> if you see this NXDOMAIN "error" [outside a captive portal] it is almost always just a normal lookup of a domain that doesn't exist,

... which is for sure very rare on my (admittedly headless) system. Not sure why querying inexistent domains would be frequent on others' systems?

> and the error message is simply wrong

It says "mitigating _potential_..." so it's not strictly speaking wrong. Agreed it can be misleading but summarizing this fairly complex issue in just one line is quite a challenge.

> This is due to a Ubuntu-only patch to systemd

Care to share the "best" link(s) about this? I mean links to best bugs and/or relevant source code. An internet search returns too many hits! BTW a good and authoritative bug number in the error message would probably not hurt either, see why above.

> network-manager could work around this problematic Ubuntu-only systemd patch,

With an Ubuntu-only network-manager patch maybe? Sorry couldn't resist :-)

I ran tcpdump 'host ubuntu20 and port domain' on my router and looked at all the other NXDOMAIN at boot time besides this one.

1. PTR request on the IPv4 address of my virbr0 interface. Gone after uninstalling a couple libvirt-* packages

2. PTR request on ONE of my IPv6 addresses. It's not obvious to me which address gets picked for that PTR query. I disabled IPv6 ULA on my router and the PTR request just moved to a different address.

3. SOA local. ??

I also found that NetworkManager needs to be told "connectivity-check.ubuntu.com has no AAAA record" about 12 times during boot, that seems a bit excessive? (this is not NXDOMAIN)

I just upgraded two machines to Ubuntu 20.04 LTS and /var/log/syslog is full of:

systemd-resolved[...]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

with the only thing reported by tcpdump(8) as:

[...] [udp sum ok] 9797 NXDomain* q: AAAA? connectivity-check.ubuntu.com.[...]

The only thing that fixes it is by making connectivity-check.ubuntu.com fully qualified.

I have over 250 machines to upgrade. It all adds up.

Please append the dot.

Hello MarcH, or anyone else affected,

Accepted systemd into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/246.6-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello MarcH, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello MarcH, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.45 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (237-3ubuntu10.45) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

corosync/2.4.3-0ubuntu1.1 (armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (amd64)
openssh/1:7.6p1-4ubuntu0.3 (amd64, ppc64el, arm64, i386, armhf, s390x)
linux-hwe-5.0/5.0.0-65.71 (i386)
linux-hwe-5.4/5.4.0-67.75~18.04.1 (i386)
gvfs/1.36.1-0ubuntu1.3.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.5) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus-apache-exporter/0.7.0+ds-1 (arm64)
udisks2/2.8.4-1ubuntu1 (amd64, ppc64el)
munin/2.0.56-1ubuntu1 (arm64)
docker.io/19.03.8-0ubuntu1.20.04.2 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted systemd (246.6-1ubuntu1.2) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

snapd/2.48.3+20.10 (amd64)
nut/2.7.4-12ubuntu1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

with -release version of systemd:

root@lp1880258-g:~# host afhjisrfljsdirfj.com
Host afhjisrfljsdirfj.com not found: 3(NXDOMAIN)

root@lp1880258-g:~# journalctl -b -u systemd-resolved --no-pager
...
Mar 09 18:07:46 lp1880258-g systemd-resolved[624]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mar 09 18:09:11 lp1880258-g systemd-resolved[624]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mar 09 18:09:14 lp1880258-g systemd-resolved[624]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

root@lp1880258-g:~# dpkg -l systemd|grep systemd
ii systemd 246.6-1ubuntu1.2 amd64 system and service manager
root@lp1880258-g:~# host afhjisrfljsdirfj.com
Host afhjisrfljsdirfj.com not found: 3(NXDOMAIN)
root@lp1880258-g:~# host afhjisrfljsdirfj.com
Host afhjisrfljsdirfj.com not found: 3(NXDOMAIN)
root@lp1880258-g:~# host afhjisrfljsdirfj.com
Host afhjisrfljsdirfj.com not found: 3(NXDOMAIN)
root@lp1880258-g:~# journalctl -b -u systemd-resolved --no-pager | grep NXDOMAIN
root@lp1880258-g:~#

root@lp1880258-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.4 amd64 system and service manager
root@lp1880258-f:~# host gfjrisldfjrfj.com
Host gfjrisldfjrfj.com not found: 3(NXDOMAIN)
root@lp1880258-f:~# journalctl -b -u systemd-resolved | grep NXDOMAIN
Mar 09 18:15:29 lp1880258-f systemd-resolved[624]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mar 09 18:24:39 lp1880258-f systemd-resolved[624]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mar 09 18:24:49 lp1880258-f systemd-resolved[624]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

(upgrade and reboot)

root@lp1880258-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.5 amd64 system and service manager
root@lp1880258-f:~# host gfjrisldfjrfj.com
Host gfjrisldfjrfj.com not found: 3(NXDOMAIN)
root@lp1880258-f:~# journalctl -b -u systemd-resolved | grep NXDOMAIN
root@lp1880258-f:~#

root@lp1880258-b:~# dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.44 amd64 system and service manager
root@lp1880258-b:~# host jifaslrjfiudrlj.com
Host jifaslrjfiudrlj.com not found: 3(NXDOMAIN)
root@lp1880258-b:~# journalctl -b -u systemd-resolved | grep NXDOMAIN
Mar 09 18:15:29 lp1880258-b systemd-resolved[786]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Mar 09 18:25:51 lp1880258-b systemd-resolved[786]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

(upgrade/reboot)

root@lp1880258-b:~# dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.45 amd64 system and service manager
root@lp1880258-b:~# host jifaslrjfiudrlj.com
Host jifaslrjfiudrlj.com not found: 3(NXDOMAIN)
root@lp1880258-b:~# journalctl -b -u systemd-resolved | grep NXDOMAIN
root@lp1880258-b:~#

fixed in hirsute with:
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=299002546ec2d62e7f0dd7d614ba958fc9df83c2

at version 247.3-1ubuntu4

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 246.6-1ubuntu1.2

---------------
systemd (246.6-1ubuntu1.2) groovy; urgency=medium

  [ Ioanna Alifieraki ]
  * d/p/lp1911187-systemctl-do-not-shutdown-immediately-on-scheduled-shutdo.patch:
    Do not shutdown immediately when scheduled shutdown fails (LP: #1911187)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=bbca658178d27e0ffaf759ba3115878d26c917a6

  [ Dan Streetman ]
  * d/p/debian/patches/lp1880258-log-nxdomain-as-debug.patch:
    Change NXDOMAIN 'errors' to log level debug (LP: #1880258)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b8e864eb907469616d688eab39b20491693d7c3a
  * d/p/lp1913763-udev-rules-add-rule-to-create-dev-ptp_hyperv.patch:
    Create symlink for hyperv-provided ptp device (LP: #1913763)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=83c7476c80e4139ee7c1ec18a549c30d37d3b4b2

 -- Ioanna Alifieraki <email address hidden> Tue, 23 Feb 2021 02:31:55 +0200

This bug was fixed in the package systemd - 245.4-4ubuntu3.5

---------------
systemd (245.4-4ubuntu3.5) focal; urgency=medium

  [ Ioanna Alifieraki ]
  * d/p/lp1911187-systemctl-do-not-shutdown-immediately-on-scheduled-shutdo.patch:
    Do not shutdown immediately when scheduled shutdown fails (LP: #1911187)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3899c9d5c171e84fc503c6ab46aea7cc9def7235

  [ Dimitri John Ledkov ]
  * d/p/lp1878969-meson-initialize-time-epoch-to-reproducible-builds-compat.patch:
    meson: initialize time-epoch to reproducible builds compatible value
    (LP: #1878969)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=84212797d21ed08031e1d71fe5e118bdd9873c0f

  [ Dan Streetman ]
  * d/p/lp1913189-test-accept-that-char-device-0-0-can-now-be-created-.patch:
    - Fix failing test case under 5.8 kernel (LP: #1913189)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=782a382017ce188dbf9a40adabd265943d7db119
  * d/p/lp1913423-hashmap-make-sure-to-initialize-shared-hash-key-atom.patch:
    Thread-safe init of hashmap shared key (LP: #1913423)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=fc8dde7c4b1944d1583866f61c2314174b4dd06a
  * d/p/lp1902236-nss-systemd-don-t-synthesize-root-nobody-when-iterat.patch:
    Don't synthesize root/nobody when iterating (LP: #1902236)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4d20e415ecd5b0fd032b4cf45bd9fd344cc434ac
  * d/p/debian/patches/lp1880258-log-nxdomain-as-debug.patch:
    Change NXDOMAIN 'errors' to log level debug (LP: #1880258)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=80163a2097aa876fe73b6071495ae4ad8749d04e
  * d/p/lp1913763-udev-rules-add-rule-to-create-dev-ptp_hyperv.patch:
    Create symlink for hyperv-provided ptp device (LP: #1913763)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b02053da2ff3fee6221bd8310488af0f52b140f1

 -- Ioanna Alifieraki <email address hidden> Tue, 23 Feb 2021 00:18:57 +0000

This bug was fixed in the package systemd - 237-3ubuntu10.45

---------------
systemd (237-3ubuntu10.45) bionic; urgency=medium

  [ Ioanna Alifieraki ]
  * d/p/lp1911187-systemctl-do-not-shutdown-immediately-on-scheduled-shutdo.patch:
    Do not shutdown immediately when scheduled shutdown fails (LP: #1911187)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=257135a59455f4e4063e78cdd3f5cfeca2597b5b

  [ Dimitri John Ledkov ]
  * d/p/lp1878969-meson-initialize-time-epoch-to-reproducible-builds-compat.patch:
    meson: initialize time-epoch to reproducible builds compatible value
    (LP: #1878969)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6f5a0c94ff4a486ee0b72af926672b24d16ff5a8

  [ Dan Streetman ]
  * d/p/lp1913189-test-accept-that-char-device-0-0-can-now-be-created-.patch:
    - Fix failing test case under 5.8 kernel (LP: #1913189)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=15143ec6cd584a18866390a042348a543e5aa22d
  * d/p/lp1913423-hashmap-make-sure-to-initialize-shared-hash-key-atom.patch:
    Thread-safe init of hashmap shared key (LP: #1913423)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=95c189adb9c3e22576b26b084c7edf001cbc8307
  * d/p/lp1890448-hwdb-Add-EliteBook-to-use-micmute-hotkey.patch:
    Add EliteBook to use micmute hotkey (LP: #1890448)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19b48bdac5129aa772fbcd2dbf8d1bb5c30c1510
  * d/p/debian/patches/lp1902553-test-disable-QEMU-based-testing-for-TEST-16-EXTEND-T.patch:
    Disable TEST-03 run under qemu (LP: #1902553)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4e37d20ec379d169cfd53088d0c3b4d7bb65d25b
  * d/p/debian/patches/lp1883447-seccomp-add-all-time64-syscalls.patch:
    Add *time64 syscalls (LP: #1883447)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a459492c67c5c5855b03daca4b44141705495376
  * d/p/lp1685754-pid1-by-default-make-user-units-inherit-their-umask-.patch:
    Inherit umask for --user processes (LP: #1685754)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=00df8d0e76975594adb765182c587ef495262fe1
  * d/p/debian/patches/lp1880258-log-nxdomain-as-debug.patch:
    Change NXDOMAIN 'errors' to log level debug (LP: #1880258)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9684abed02669bfcf696763b887518cf54cd3f69
  * d/p/lp1913763-udev-rules-add-rule-to-create-dev-ptp_hyperv.patch:
    Create symlink for hyperv-provided ptp device (LP: #1913763)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ff2a9ed2ece6bbd86a3d57f42b26cb1a6ca2845a

 -- Ioanna Alifieraki <email address hidden> Tue, 23 Feb 2021 03:45:01 +0200