MAAS

dnssec failures cause nodes to be unable to resolve external addresses

Bug #1384334 reported by Jeff Lane  on 2014-10-22

This bug affects 9 people

Affects		Status	Importance	Assigned to	Milestone
	MAAS	Fix Released	Critical	Mike Pontillo	MAAS 1.8.0

Bug Description

https://pastebin.canonical.com/119180/

MAAS 1.5.4 running on Trusty, fresh default install, the only things we've done are standard config (setting up network, adding DNS forwarder address, and then adding nodes once images were synced).

The problem was that I was unable to resolve addresses using the MAAS DNS forwarder. For example:
ubuntu@utsa-maas:/etc/bind$ dig @192.168.1.1 streams.canonical.com

; <<>> DiG 9.9.5-3-Ubuntu <<>> @192.168.1.1 streams.canonical.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 10035
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;streams.canonical.com. IN A

;; Query time: 0 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Wed Oct 22 11:32:29 CDT 2014
;; MSG SIZE rcvd: 50

I would get that, and a corresponding error in /var/log/syslog:
Oct 22 11:18:20 utsa-maas named[41376]: error (insecurity proof failed) resolving './NS/IN': 10.241.1.1#53

I believe the real cause is that dnssec-validation is set to auto by default in bind9, and those dnssec queries failed, which caused nodes in the MAAS cluster to be unable to resolve external addresses. This, in turn, caused juju bootstrap to fail:

Setting up libgoogle-perftools4 (2.1-2ubuntu1) ...
Setting up libsnappy1 (1.1.0-1ubuntu1) ...
Setting up juju-mongodb (2.4.9-0ubuntu3) ...
Processing triggers for libc-bin (2.19-0ubuntu6.3) ...
curl: (6) Could not resolve host: streams.canonical.com
tools from https://streams.canonical.com/juju/tools/releases/juju-1.18.4-trusty-amd64.tgz downloaded: HTTP 000; time 9.519s; size 0 bytes; speed 0.000 bytes/s ERROR bootstrap failed: rc: 1
Stopping instance...
ERROR rc: 1

After talking about this with Kiko, I set dnssec-validation to no:

dnssec-validation no;

in /etc/bind/named.conf.options

and after restarting bind, external forwarding started working properly.

Tags:

Related branches

lp:~mpontillo/maas/bug-1384334-add-dnssec-config-option

Merged into lp:~maas-committers/maas/trunk at revision 3852

Gavin Panella (community): Approve on 2015-04-28

lp:~mpontillo/maas/packaging-dnssec-validation-fix

Merged into lp:~maas-maintainers/maas/packaging at revision 388

Gavin Panella (community): Approve on 2015-04-29

Revision history for this message

Andres Rodriguez (andreserl) wrote on 2014-10-22:

Did you add upstream DNS in MAAS settings page?

Revision history for this message

Jeff Lane  (bladernr) wrote on 2014-10-22:

Yes... on the settings page I added in UTSA's DNS server. This was confirmed in the options file:

ubuntu@utsa-maas:/etc/bind/maas$ cat named.conf.options.inside.maas
    forwarders {
        10.241.1.1;
    };

Julian Edwards (julian-edwards) on 2014-10-22

Changed in maas:
status:	New → Triaged
importance:	Undecided → High
tags:	added: dns trivial
Changed in maas:
milestone:	none → next

Revision history for this message

Andres Rodriguez (andreserl) wrote on 2014-10-27:

So there's a few people who have hit this issue... I think it would probably be a good idea to add an option to either enable/disable DNSSEC validation on the config.. what do you think?

Revision history for this message

Narinder Gupta (narindergupta) wrote on 2014-10-29:

even at HP environmment i have top change it to dnssec-validation no; to make it work from day 1.

Revision history for this message

Graham Binns (gmb) wrote on 2014-11-07:

Julian, you've marked this as trivial, but it's not clear why. Do you think this is just a template change (as opposed to Andres's suggestion of a config option)?

Revision history for this message

Julian Edwards (julian-edwards) wrote on 2014-11-09:

Graham, either way it's trivial. The Config stuff makes it easy.

Andres Rodriguez (andreserl) on 2015-04-10

Changed in maas:
milestone:	next → 1.8.0
importance:	High → Critical

Mike Pontillo (mpontillo) on 2015-04-22

Changed in maas:
assignee:	nobody → Mike Pontillo (mpontillo)

Mike Pontillo (mpontillo) on 2015-04-24

Changed in maas:
status:	Triaged → In Progress

MAAS Lander (maas-lander) on 2015-04-29

Changed in maas:
status:	In Progress → Fix Committed

Andres Rodriguez (andreserl) on 2015-06-22

Changed in maas:
status:	Fix Committed → Fix Released

Revision history for this message

Alexander List (alexlist) wrote on 2015-08-21:

This affects the current MAAS delivered with trusty (1.7) - can we expect a SRU?

Revision history for this message

Mike Pontillo (mpontillo) wrote on 2015-08-21:

We added the option to the MAAS settings page as a convenience for users, since we also allow users to specify forwarders there. (it made sense that, when specifying forwarders, you should consider whether or not those forwarders support DNSSEC.)

Since this wasn't really a MAAS problem (rather, a BIND configuration option that MAAS will now manage for users), I'm not sure an SRU can be justified. So I suggest using the workaround in the description. That is, set the following option in /etc/bind/named.conf.options:

dnssec-validation no;

Either that, or just use MAAS 1.8 out the gate. (it's in the 'stable' PPA.) Note that later when you upgrade to MAAS 1.8+, we'll attempt to parse the existing named.conf.options and migrate your DNSSEC setting to the MAAS database.

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.