Bug #1751797 “NetworkManager incorrectly uses resolved' route-on...” : Bugs : systemd package : Ubuntu

Revision history for this message

Scott Moser (smoser) wrote on 2018-02-26:

#1

output of systemd-resolve --status Edit (3.0 KiB, text/plain)
CurrentDmesg.txt Edit (72.5 KiB, text/plain; charset="utf-8")
Dependencies.txt Edit (3.1 KiB, text/plain; charset="utf-8")
HookError_systemd.txt Edit (999 bytes, text/plain; charset="utf-8")
JournalErrors.txt Edit (4.1 KiB, text/plain; charset="utf-8")
Lspci.txt Edit (8.8 KiB, text/plain; charset="utf-8")
Lsusb.txt Edit (436 bytes, text/plain; charset="utf-8")
ProcCpuinfo.txt Edit (4.4 KiB, text/plain; charset="utf-8")
ProcCpuinfoMinimal.txt Edit (1.1 KiB, text/plain; charset="utf-8")
ProcInterrupts.txt Edit (2.5 KiB, text/plain; charset="utf-8")
ProcModules.txt Edit (9.7 KiB, text/plain; charset="utf-8")
UdevDb.txt Edit (182.1 KiB, text/plain; charset="utf-8")

Revision history for this message

Scott Moser (smoser) wrote on 2018-02-26:

#2

I had hoped that simply disconnect/reconnect of the interface would work, but that did not improve things. (nmcli device disconnect enp0s25 ; nmcli device connect enp0s25)

Scott Moser (smoser) on 2018-02-26

description:

updated

Revision history for this message

Launchpad Janitor (janitor) wrote on 2018-02-26:

#3

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status:	New → Confirmed

Revision history for this message

Dan Watkins (oddbloke) wrote on 2018-02-26:

#4

$ nmcli device show wlp3s0 | pastebinit
http://paste.ubuntu.com/p/xJ6YjD8WQ9/

$ systemd-resolve --status | pastebinit
http://paste.ubuntu.com/p/bPDMzY3sYh/

Revision history for this message

Dan Watkins (oddbloke) wrote on 2018-02-26:

#5

Oops, removed the rest of my comment when pasting; I'm also seeing this issue, but on a wireless interface.

Scott Moser (smoser) on 2018-03-02

Changed in systemd (Ubuntu):
importance:	Undecided → Critical

Revision history for this message

Scott Moser (smoser) wrote on 2018-03-15:

#6

$ systemd-resolve --status enp0s25
Link 2 (enp0s25)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.7.0.1
                      fdfd:dddd:dddd::1
          DNS Domain: ~mydomain.com

$ cat /var/lib/NetworkManager/dhclient-72c7fac3-c017-4b76-9954-b4fb08262376-enp0s25.lease
lease {
  interface "enp0s25";
  fixed-address 10.7.2.103;
  filename "pxelinux.0";
  option subnet-mask 255.255.0.0;
  option dhcp-lease-time 43200;
  option routers 10.7.0.1;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.7.0.1;
  option domain-name-servers 10.7.0.1;
  option dhcp-renewal-time 21600;
  option dhcp-rebinding-time 37800;
  option broadcast-address 10.7.255.255;
  option host-name "milhouse-eth0";
  option domain-name "mydomain.com";
  renew 4 2018/03/15 18:51:36;
  rebind 4 2018/03/15 23:41:37;
  expire 5 2018/03/16 01:11:37;
}
lease {
  interface "enp0s25";
  fixed-address 10.7.2.103;
  filename "pxelinux.0";
  option subnet-mask 255.255.0.0;
  option routers 10.7.0.1;
  option dhcp-lease-time 43200;
  option dhcp-message-type 5;
  option domain-name-servers 10.7.0.1;
  option dhcp-server-identifier 10.7.0.1;
  option dhcp-renewal-time 21600;
  option broadcast-address 10.7.255.255;
  option dhcp-rebinding-time 37800;
  option host-name "milhouse-eth0";
  option domain-name "mydomain.com";
  renew 4 2018/03/15 18:19:57;
  rebind 4 2018/03/15 23:48:56;
  expire 5 2018/03/16 01:18:56;
}

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2018-03-15:

#7

DNS Domain: ~mydomain.com means that said domain got added as "route_only" to systemd-resolved.

We need to find who/what/why did this.

This could be the configuration of resolved.conf, networkd, or network-manager itself.

Can we make sure there is nothing suspicious in:

$ grep Domains= /etc/systemd/resolved.conf \
/etc/systemd/resolved.conf.d/* \
/run/systemd/resolved.conf \
/run/systemd/resolved.conf.d/* \
/lib/systemd/resolved.conf \
/lib/systemd/resolved.conf.d/* \
/usr/lib/systemd/resolved.conf \
/usr/lib/systemd/resolved.conf.d/* \
/etc/systemd/network/* \
/etc/systemd/network/*.d/* \
/run/systemd/network/* \
/run/systemd/network/*.d/* \
/lib/systemd/network/* \
/lib/systemd/network/*.d/* \
/usr/lib/systemd/network/* \
/usr/lib/systemd/network/*.d/*

I expect loads of errors, and just
'/etc/systemd/resolved.conf:#Domains='
to turn up.

That should exclude any systemd-resolved / networkd configurations.

And I'll start digging into network-manager resolved-dns plugin code.

Revision history for this message

Scott Moser (smoser) wrote on 2018-03-15:

#8

Download full text (8.8 KiB)

connection.id: System Ethernet
connection.uuid: 72c7fac3-c017-4b76-9954-b4fb08262376
connection.stable-id: --
connection.type: 802-3-ethernet
connection.interface-name: --
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.autoconnect-retries: -1 (default)
connection.auth-retries: -1
connection.timestamp: 1521122033
connection.read-only: no
connection.permissions: --
connection.zone: --
connection.master: --
connection.slave-type: --
connection.autoconnect-slaves: -1 (default)
connection.secondaries: --
connection.gateway-ping-timeout: 0
connection.metered: unknown
connection.lldp: default
802-3-ethernet.port: --
802-3-ethernet.speed: 0
802-3-ethernet.duplex: --
802-3-ethernet.auto-negotiate: no
802-3-ethernet.mac-address: --
802-3-ethernet.cloned-mac-address: --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist: --
802-3-ethernet.mtu: auto
802-3-ethernet.s390-subchannels: --
802-3-ethernet.s390-nettype: --
802-3-ethernet.s390-options: --
802-3-ethernet.wake-on-lan: default
802-3-ethernet.wake-on-lan-password: --
ipv4.method: auto
ipv4.dns: --
ipv4.dns-search: --
ipv4.dns-options: ""
ipv4.dns-priority: 0
ipv4.addresses: --
ipv4.gateway: --
ipv4.routes: --
ipv4.route-metric: -1
ipv4.route-table: 0 (unspec)
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-timeout: 0 (default)
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.dhcp-fqdn: --
ipv4.never-default: no
ipv4.may-fail: yes
ipv4.dad-timeout: -1 (default)
ipv6.method: auto
ipv6.dns: --
ipv6.dns-search: --
ipv6.dns-options: ""
ipv6.dns-priority: 0
ipv6.addresses: --
ipv6.gateway: --
ipv6.routes: --
ipv6.route-metric: -1
ipv6.route-table: 0 (unspec)
ipv6.ignore-auto-routes: no
ipv6.ignore-auto-dns: no
ipv6.never-default: no
ipv6.may-fail: yes
ipv6.ip6-privacy: -1 (unknown)
ipv6.addr-gen-mode: stable-privacy
ipv6.dhcp-send-hostname: ...

connection.id:                          System Ethernet
connection.uuid:                        72c7fac3-c017-4b76-9954-b4fb08262376
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              --
connection.autoconnect:                 yes
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.auth-retries:                -1
connection.timestamp:                   1521122033
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             --
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
ipv4.method:                            auto
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       ""
ipv4.dns-priority:                      0
ipv4.addresses:                         --
ipv4.gateway:                           --
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv6.method:                            auto
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       ""
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.token:                             --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           System Ethernet
GENERAL.UUID:                           72c7fac3-c017-4b76-9954-b4fb08262376
GENERAL.DEVICES:                        enp0s25
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        yes
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/1
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         10.7.2.103/16
IP4.GATEWAY:                            10.7.0.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 10.7.0.1, mt = 100
IP4.ROUTE[2]:                           dst = 10.7.0.0/16, nh = 0.0.0.0, mt = 100
IP4.ROUTE[3]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000
IP4.DNS[1]:                             10.7.0.1
IP4.DOMAIN[1]:                          mydomain.com
DHCP4.OPTION[1]:                        requested_subnet_mask = 1
DHCP4.OPTION[2]:                        requested_rfc3442_classless_static_routes = 1
DHCP4.OPTION[3]:                        subnet_mask = 255.255.0.0
DHCP4.OPTION[4]:                        domain_name_servers = 10.7.0.1
DHCP4.OPTION[5]:                        ip_address = 10.7.2.103
DHCP4.OPTION[6]:                        filename = pxelinux.0
DHCP4.OPTION[7]:                        requested_static_routes = 1
DHCP4.OPTION[8]:                        dhcp_server_identifier = 10.7.0.1
DHCP4.OPTION[9]:                        requested_time_offset = 1
DHCP4.OPTION[10]:                       broadcast_address = 10.7.255.255
DHCP4.OPTION[11]:                       requested_interface_mtu = 1
DHCP4.OPTION[12]:                       dhcp_rebinding_time = 37800
DHCP4.OPTION[13]:                       requested_domain_name_servers = 1
DHCP4.OPTION[14]:                       dhcp_message_type = 5
DHCP4.OPTION[15]:                       requested_broadcast_address = 1
DHCP4.OPTION[16]:                       routers = 10.7.0.1
DHCP4.OPTION[17]:                       dhcp_renewal_time = 21600
DHCP4.OPTION[18]:                       requested_domain_name = 1
DHCP4.OPTION[19]:                       domain_name = mydomain.com
DHCP4.OPTION[20]:                       requested_routers = 1
DHCP4.OPTION[21]:                       expiry = 1521163136
DHCP4.OPTION[22]:                       requested_wpad = 1
DHCP4.OPTION[23]:                       requested_netbios_scope = 1
DHCP4.OPTION[24]:                       host_name = milhouse-eth0
DHCP4.OPTION[25]:                       requested_ms_classless_static_routes = 1
DHCP4.OPTION[26]:                       requested_netbios_name_servers = 1
DHCP4.OPTION[27]:                       network_number = 10.7.0.0
DHCP4.OPTION[28]:                       requested_domain_search = 1
DHCP4.OPTION[29]:                       next_server = 10.7.10.1
DHCP4.OPTION[30]:                       requested_ntp_servers = 1
DHCP4.OPTION[31]:                       requested_host_name = 1
DHCP4.OPTION[32]:                       dhcp_lease_time = 43200
IP6.ADDRESS[1]:                         fdfd:dddd:dddd::9fa/128
IP6.ADDRESS[2]:                         fdfd:dddd:dddd:0:20f8:fe9c:5b0b:14d5/64
IP6.ADDRESS[3]:                         fdfd:dddd:dddd:0:4aeb:70e7:3e74:10c5/64
IP6.ADDRESS[4]:                         fe80::64f4:6bde:246e:6d80/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fdfd:dddd:dddd::/48, nh = fe80::c24a:ff:fedd:541e, mt = 100
IP6.ROUTE[2]:                           dst = fdfd:dddd:dddd::/64, nh = ::, mt = 100
IP6.ROUTE[3]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[4]:                           dst = fe80::/64, nh = ::, mt = 256
IP6.ROUTE[5]:                           dst = fe80::/64, nh = ::, mt = 100
IP6.ROUTE[6]:                           dst = fdfd:dddd:dddd::9fa/128, nh = ::, mt = 100
IP6.DNS[1]:                             fdfd:dddd:dddd::1
DHCP6.OPTION[1]:                        dhcp6_client_id = 0:4:b6:a3:4e:94:10:5a:74:69:20:d6:3e:4:14:31:a7:63
DHCP6.OPTION[2]:                        dhcp6_unknown_11 = 3:1:0:5a:aa:72:c4:0:0:0:82:1:bc:31:b9:64:a2:c2:f1:d3:a9:5d:fe:ae:97:e5:60:c9
DHCP6.OPTION[3]:                        requested_dhcp6_name_servers = 1
DHCP6.OPTION[4]:                        dhcp6_name_servers = fdfd:dddd:dddd::1
DHCP6.OPTION[5]:                        rebind = 34560
DHCP6.OPTION[6]:                        max_life = 4294967295
DHCP6.OPTION[7]:                        preferred_life = 4294967295
DHCP6.OPTION[8]:                        requested_dhcp6_domain_search = 1
DHCP6.OPTION[9]:                        requested_dhcp6_client_id = 1
DHCP6.OPTION[10]:                       life_starts = 1521119940
DHCP6.OPTION[11]:                       ip6_address = fdfd:dddd:dddd::9fa
DHCP6.OPTION[12]:                       ip6_prefixlen = 64
DHCP6.OPTION[13]:                       dhcp6_domain_search = mydomain.com.
DHCP6.OPTION[14]:                       dhcp6_solmax_rt = 60
DHCP6.OPTION[15]:                       renew = 21600
DHCP6.OPTION[16]:                       dhcp6_server_id = 0:3:0:1:c0:4a:0:dd:54:1e
DHCP6.OPTION[17]:                       iaid = ed:75:5f:9a
DHCP6.OPTION[18]:                       starts = 1521119940

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2018-03-15:

#9

Fixed upstream in https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=e91f1a7d2a6b8400b6b331d5b72287dcb5164a39

summary:	- dns resolution only works for domains in 'search'. + NetworkManager incorrectly uses resolved' route-only DNS setting, + resulting in dns resolution only working for domains in 'search'.
Changed in systemd (Ubuntu):
status:	Confirmed → Invalid

Dimitri John Ledkov (xnox) on 2018-03-15

Changed in network-manager (Ubuntu):
status:	New → Fix Committed

Scott Moser (smoser) on 2018-03-15

Changed in network-manager (Ubuntu):
importance:	Undecided → Critical

Revision history for this message

Launchpad Janitor (janitor) wrote on 2018-03-17:

#10

This bug was fixed in the package network-manager - 1.10.4-1ubuntu3

---------------
network-manager (1.10.4-1ubuntu3) bionic; urgency=medium

  * Cherry-pick a partial patch from 1.11 branch, to stop agressively
    setting route_only flag on nameservers in the systemd-resolved dns
    plugin. LP: #1751797

-- Dimitri John Ledkov <email address hidden> Thu, 15 Mar 2018 15:24:01 +0000

Changed in network-manager (Ubuntu):
status:	Fix Committed → Fix Released

Ubuntu
systemd package

NetworkManager incorrectly uses resolved' route-only DNS setting, resulting in dns resolution only working for domains in 'search'.

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Affects		Status	Importance	Assigned to	Milestone
	network-manager (Ubuntu)	Fix Released	Critical	Unassigned
	systemd (Ubuntu)	Invalid	Critical	Unassigned

Ubuntusystemd package

NetworkManager incorrectly uses resolved' route-only DNS setting, resulting in dns resolution only working for domains in 'search'.

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
systemd package