Merge lp:~mpontillo/maas/lease-parser-ignore-free-leases--1.9 into lp:maas/1.9

Proposed by Mike Pontillo
Status: Merged
Approved by: Mike Pontillo
Approved revision: no longer in the source branch.
Merged at revision: 4597
Proposed branch: lp:~mpontillo/maas/lease-parser-ignore-free-leases--1.9
Merge into: lp:maas/1.9
Diff against target: 449 lines (+192/-51)
5 files modified
src/maasserver/rpc/leases.py (+2/-0)
src/maasserver/rpc/tests/test_regionservice.py (+6/-1)
src/provisioningserver/dhcp/leases_parser.py (+32/-11)
src/provisioningserver/dhcp/leases_parser_fast.py (+24/-6)
src/provisioningserver/dhcp/tests/test_leases_parser.py (+128/-33)
To merge this branch: bzr merge lp:~mpontillo/maas/lease-parser-ignore-free-leases--1.9
Reviewer Review Type Date Requested Status
Blake Rouse (community) Approve
Review via email: mp+316300@code.launchpad.net

Commit message

DHCP lease parser fixes.

 * Fix improper handling of DHCP leases with a 'free' binding state,
   and/or no 'ends' time.
 * Fix lease parser to properly parse static host mappings with a '-'
   in the mapping key (such as the entries that MAAS itself writes).
 * Fix DNS zones to be updated when lease parser finishes and updates
   the region.
 * Fix lease parser to properly remove released leases.
 * Fix traceback that could occur when the fast lease parser attempts
   interpret a released lease.

To post a comment you must log in.
Revision history for this message
Mike Pontillo (mpontillo) wrote :

The way this code works is subtle. There is a difference between a "free" DHCP lease (free because it has not been renewed) and a "released" DHCP lease. And that difference is that when you don't explicitly release the address, you see something like:

lease 172.16.99.101 {
  starts 5 2017/02/03 02:05:01;
  ends 5 2017/02/03 02:05:31;
  tstp 5 2017/02/03 02:05:31;
  cltt 5 2017/02/03 02:05:01;
  binding state free;
  hardware ethernet 52:54:00:3d:e7:29;
}

So the lease is still valid, in truth, in the above case. (because it has not been renewed.) An explicitly released address will look like this:

lease 172.16.99.102 {
  starts 5 2017/02/03 02:06:06;
  cltt 5 2017/02/03 02:06:06;
  binding state free;
  hardware ethernet 52:54:00:8f:10:eb;
}

Notice that there is no 'ends' time.

So the root cause of the bug is: a free lease without an 'ends' time was considered an 'infinite' lease. This meant MAAS confusion*, because if IP address X was leased to host 'foo' and then explicitly released (and therefore did not have an 'ends' time, because the lease was cut short), and later host 'bar' leases IP address X, it will appear that two hosts have leased the same IP address. Ouch,

*: Bad pun, sorry.

Revision history for this message
Mike Pontillo (mpontillo) wrote :

To prevent the possibility of stale leases remaining in the StaticIPAddress table as DISCOVERED, an additional fix was needed to filter stale leases (in case an IP address was assigned to another host).

One issue remains that I found in testing: after the lease parser runs, the zone file isn't udpated. (Rather, I saw it get updated ~3 seconds before the lease parser finished and uploaded the leases.) Therefore I had an out-of-date zone file even though the lease parser had already determined the correct (hostname, ip) mapping, and that mapping was shown in the UI. (Workaround: `service maas-regiond restart`.)

Revision history for this message
Blake Rouse (blake-rouse) wrote :

Looks good. As for the restart of maas-regiond. Did it never get updated? As without it updating the zone files issues will still exists resolving dynamic IP addresses.

review: Approve
Revision history for this message
Mike Pontillo (mpontillo) wrote :

Blake, thanks for taking a look. It turned out there was a separate issue with BIND that also prevented the DNS zones from being properly updated. LaMont has landed a fix (and has queued a corresponding SRU for BIND).

Revision history for this message
Blake Rouse (blake-rouse) wrote :

Awesome. Glad this is resolved. Looks good!

review: Approve
Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (1.3 MiB)

The attempt to merge lp:~mpontillo/maas/lease-parser-ignore-free-leases--1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

Ign http://prodstack-zone-1.clouds.archive.ubuntu.com trusty InRelease
Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports InRelease
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release
Get:3 http://security.ubuntu.com trusty-security/main Sources [124 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main Sources [390 kB]
Get:5 http://security.ubuntu.com trusty-security/universe Sources [49.3 kB]
Get:6 http://security.ubuntu.com trusty-security/main amd64 Packages [578 kB]
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted Sources [5,911 B]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe Sources [173 kB]
Get:9 http://security.ubuntu.com trusty-security/universe amd64 Packages [151 kB]
Get:10 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse Sources [7,535 B]
Get:11 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [943 kB]
Get:12 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted amd64 Packages [16.4 kB]
Get:13 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [397 kB]
Get:14 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.0 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse amd64 Packages
Fetched 2,982 kB in 1s (1,802 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive...

Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (1.3 MiB)

The attempt to merge lp:~mpontillo/maas/lease-parser-ignore-free-leases--1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

Ign http://prodstack-zone-1.clouds.archive.ubuntu.com trusty InRelease
Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports InRelease
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release
Get:3 http://security.ubuntu.com trusty-security/main Sources [124 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main Sources [390 kB]
Get:5 http://security.ubuntu.com trusty-security/universe Sources [49.3 kB]
Get:6 http://security.ubuntu.com trusty-security/main amd64 Packages [578 kB]
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted Sources [5,911 B]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe Sources [173 kB]
Get:9 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse Sources [7,535 B]
Get:10 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [943 kB]
Get:11 http://security.ubuntu.com trusty-security/universe amd64 Packages [151 kB]
Get:12 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted amd64 Packages [16.4 kB]
Get:13 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [397 kB]
Get:14 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.0 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse amd64 Packages
Fetched 2,982 kB in 1s (1,718 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive...

Revision history for this message
Mike Pontillo (mpontillo) wrote :

Hm. I don't understand why this keeps failing. I just SSH'd to the lander and tried the same test, and it passed.

Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (1.4 MiB)

The attempt to merge lp:~mpontillo/maas/lease-parser-ignore-free-leases--1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Get:2 http://security.ubuntu.com trusty-security/main Sources [124 kB]
Get:3 http://security.ubuntu.com trusty-security/universe Sources [49.3 kB]
Get:4 http://security.ubuntu.com trusty-security/main amd64 Packages [578 kB]
Get:5 http://security.ubuntu.com trusty-security/universe amd64 Packages [151 kB]
Ign http://prodstack-zone-1.clouds.archive.ubuntu.com trusty InRelease
Get:6 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports InRelease
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main Sources [390 kB]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted Sources [5,911 B]
Get:9 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe Sources [173 kB]
Get:10 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse Sources [7,535 B]
Get:11 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [943 kB]
Get:12 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted amd64 Packages [16.4 kB]
Get:13 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [397 kB]
Get:14 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.0 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse amd64 Packages
Fetched 2,982 kB in 2s (1,104 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'src/maasserver/rpc/leases.py'
--- src/maasserver/rpc/leases.py 2015-08-27 16:07:30 +0000
+++ src/maasserver/rpc/leases.py 2017-02-07 17:35:36 +0000
@@ -16,6 +16,7 @@
16 "update_leases",16 "update_leases",
17]17]
1818
19from maasserver.dns.config import dns_update_all_zones
19from maasserver.models.nodegroup import NodeGroup20from maasserver.models.nodegroup import NodeGroup
20from maasserver.models.staticipaddress import StaticIPAddress21from maasserver.models.staticipaddress import StaticIPAddress
21from maasserver.utils.orm import transactional22from maasserver.utils.orm import transactional
@@ -49,4 +50,5 @@
49 else:50 else:
50 leases = convert_mappings_to_leases(mappings)51 leases = convert_mappings_to_leases(mappings)
51 StaticIPAddress.objects.update_leases(nodegroup, leases)52 StaticIPAddress.objects.update_leases(nodegroup, leases)
53 dns_update_all_zones()
52 return {}54 return {}
5355
=== modified file 'src/maasserver/rpc/tests/test_regionservice.py'
--- src/maasserver/rpc/tests/test_regionservice.py 2015-11-06 16:27:35 +0000
+++ src/maasserver/rpc/tests/test_regionservice.py 2017-02-07 17:35:36 +0000
@@ -57,6 +57,7 @@
57from maasserver.models.nodegroup import NodeGroup57from maasserver.models.nodegroup import NodeGroup
58from maasserver.rpc import (58from maasserver.rpc import (
59 events as events_module,59 events as events_module,
60 leases as leases_module,
60 regionservice,61 regionservice,
61)62)
62from maasserver.rpc.regionservice import (63from maasserver.rpc.regionservice import (
@@ -84,6 +85,7 @@
84from maasserver.utils.threads import deferToDatabase85from maasserver.utils.threads import deferToDatabase
85from maastesting.djangotestcase import DjangoTransactionTestCase86from maastesting.djangotestcase import DjangoTransactionTestCase
86from maastesting.matchers import (87from maastesting.matchers import (
88 MockCalledOnce,
87 MockCalledOnceWith,89 MockCalledOnceWith,
88 MockCallsMatch,90 MockCallsMatch,
89 MockNotCalled,91 MockNotCalled,
@@ -424,9 +426,11 @@
424426
425 @wait_for_reactor427 @wait_for_reactor
426 @inlineCallbacks428 @inlineCallbacks
427 def test__stores_leases(self):429 def test__stores_leases_and_updates_dns(self):
430 dns_update = self.patch(leases_module, 'dns_update_all_zones')
428 interface, ngi, nodegroup = yield deferToDatabase(431 interface, ngi, nodegroup = yield deferToDatabase(
429 self.make_interface_on_managed_cluster_interface)432 self.make_interface_on_managed_cluster_interface)
433
430 mapping = {434 mapping = {
431 "ip": ngi.ip_range_low,435 "ip": ngi.ip_range_low,
432 "mac": interface.mac_address.get_raw(),436 "mac": interface.mac_address.get_raw(),
@@ -436,6 +440,7 @@
436 b"uuid": nodegroup.uuid, b"mappings": [mapping]})440 b"uuid": nodegroup.uuid, b"mappings": [mapping]})
437441
438 self.assertThat(response, Equals({}))442 self.assertThat(response, Equals({}))
443 self.assertThat(dns_update, MockCalledOnce())
439444
440 [(ip, mac)] = yield deferToDatabase(445 [(ip, mac)] = yield deferToDatabase(
441 self.get_leases_for, nodegroup=nodegroup)446 self.get_leases_for, nodegroup=nodegroup)
442447
=== modified file 'src/provisioningserver/dhcp/leases_parser.py'
--- src/provisioningserver/dhcp/leases_parser.py 2016-05-11 18:50:52 +0000
+++ src/provisioningserver/dhcp/leases_parser.py 2017-02-07 17:35:36 +0000
@@ -38,7 +38,9 @@
38)38)
3939
4040
41ip = Regex("[:0-9a-fA-F][:.0-9a-fA-F]{2,38}")41LEASE_TIME_FORMAT = '%w %Y/%m/%d %H:%M:%S'
42
43ip = Regex("[:0-9a-fA-F][-:.0-9a-fA-F]{2,38}")
42mac = Regex("[0-9a-fA-F]{2}(:[0-9a-fA-F]{2}){5}")44mac = Regex("[0-9a-fA-F]{2}(:[0-9a-fA-F]{2}){5}")
43hardware_type = Regex('[A-Za-z0-9_-]+')45hardware_type = Regex('[A-Za-z0-9_-]+')
44args = Regex('[^"{;]+') | QuotedString('"')46args = Regex('[^"{;]+') | QuotedString('"')
@@ -125,11 +127,20 @@
125 expiry, or None if the lease has no expiry date.127 expiry, or None if the lease has no expiry date.
126 """128 """
127 assert is_lease(lease)129 assert is_lease(lease)
128 ends = getattr(lease, 'ends', None)130 end_time = getattr(lease, 'ends', '')
129 if ends is None or len(ends) == 0 or ends.lower() == 'never':131 if end_time is None or len(end_time) == 0:
132 binding_state = getattr(lease, 'binding state', '')
133 binding_state = binding_state.lower()
134 if binding_state == 'free':
135 # For a 'free' lease, the release time is the 'starts' time.
136 start_time = getattr(lease, 'starts', '')
137 if start_time is None or len(start_time) == 0:
138 return None
139 return datetime.strptime(start_time, LEASE_TIME_FORMAT)
140 elif end_time.lower() == 'never':
130 return None141 return None
131 else:142 else:
132 return datetime.strptime(ends, '%w %Y/%m/%d %H:%M:%S')143 return datetime.strptime(end_time, LEASE_TIME_FORMAT)
133144
134145
135def has_expired(lease, now):146def has_expired(lease, now):
@@ -153,13 +164,17 @@
153def gather_leases(hosts_and_leases):164def gather_leases(hosts_and_leases):
154 """Find current leases among `hosts_and_leases`."""165 """Find current leases among `hosts_and_leases`."""
155 now = datetime.utcnow()166 now = datetime.utcnow()
156 # If multiple leases for the same address are valid at the same167 # Ensure we have the most recent MAC leased to each IP address.
157 # time, for whatever reason, the list will contain all of them.168 leases = OrderedDict()
158 return [169 for lease in filter(is_lease, hosts_and_leases):
159 (lease.host, lease.hardware.mac)170 lease_mac = get_lease_mac(lease)
160 for lease in filter(is_lease, hosts_and_leases)171 lease_ip = lease.host
161 if not has_expired(lease, now)172 if not has_expired(lease, now) and lease_mac is not None:
162 ]173 leases[lease_ip] = lease_mac
174 else:
175 if lease_ip in leases:
176 del leases[lease_ip]
177 return [(ip, mac) for ip, mac in leases.items()]
163178
164179
165def get_host_mac(host):180def get_host_mac(host):
@@ -173,6 +188,12 @@
173 return None188 return None
174 else:189 else:
175 return host190 return host
191 # In this case, it's stored the same way a lease MAC is stored.
192 return get_lease_mac(host)
193
194
195def get_lease_mac(host):
196 """Get the MAC address from a lease declaration."""
176 hardware = getattr(host, 'hardware', None)197 hardware = getattr(host, 'hardware', None)
177 if hardware in (None, '', b''):198 if hardware in (None, '', b''):
178 return None199 return None
179200
=== modified file 'src/provisioningserver/dhcp/leases_parser_fast.py'
--- src/provisioningserver/dhcp/leases_parser_fast.py 2015-09-09 15:02:30 +0000
+++ src/provisioningserver/dhcp/leases_parser_fast.py 2017-02-07 17:35:36 +0000
@@ -23,7 +23,10 @@
23 'parse_leases',23 'parse_leases',
24 ]24 ]
2525
26from collections import defaultdict26from collections import (
27 defaultdict,
28 OrderedDict,
29)
27from datetime import datetime30from datetime import datetime
28from itertools import chain31from itertools import chain
29import re32import re
@@ -31,6 +34,7 @@
31from provisioningserver.dhcp.leases_parser import (34from provisioningserver.dhcp.leases_parser import (
32 get_host_ip,35 get_host_ip,
33 get_host_mac,36 get_host_mac,
37 get_lease_mac,
34 has_expired,38 has_expired,
35 is_host,39 is_host,
36 is_lease,40 is_lease,
@@ -43,7 +47,7 @@
43 ^\s* # Ignore leading whitespace on each line.47 ^\s* # Ignore leading whitespace on each line.
44 (host|lease) # Look only for host or lease stanzas.48 (host|lease) # Look only for host or lease stanzas.
45 \s+ # Mandatory whitespace.49 \s+ # Mandatory whitespace.
46 ([0-9a-fA-F.:]+) # Capture the IP/MAC address for this stanza.50 ([0-9a-fA-F.:-]+) # Capture the IP/MAC address for this stanza.
47 \s*{ # Optional whitespace then an opening brace.51 \s*{ # Optional whitespace then an opening brace.
48 ''',52 ''',
49 re.MULTILINE | re.DOTALL | re.VERBOSE)53 re.MULTILINE | re.DOTALL | re.VERBOSE)
@@ -71,15 +75,29 @@
7175
7276
73def parse_leases(leases_contents):77def parse_leases(leases_contents):
74 results = []78 leases = OrderedDict()
79 hosts = []
75 now = datetime.utcnow()80 now = datetime.utcnow()
76 for entry in extract_leases(leases_contents):81 for entry in extract_leases(leases_contents):
77 if is_lease(entry):82 if is_lease(entry):
78 if not has_expired(entry, now):83 mac = get_lease_mac(entry)
79 results.append((entry.host, entry.hardware.mac))84 if not has_expired(entry, now) and mac is not None:
85 leases[entry.host] = entry.hardware.mac
86 else:
87 # Expired or released lease.
88 if entry.host in leases:
89 del leases[entry.host]
80 elif is_host(entry):90 elif is_host(entry):
81 mac = get_host_mac(entry)91 mac = get_host_mac(entry)
82 ip = get_host_ip(entry)92 ip = get_host_ip(entry)
83 if ip and mac:93 if ip and mac:
84 results.append((ip, mac))94 # A host entry came later than a lease entry for the same IP
95 # address. Letting them both stay will confuse MAAS by allowing
96 # an ephemeral lease to exist at the same time as a static
97 # host map.
98 if ip in leases:
99 del leases[ip]
100 hosts.append((ip, mac))
101 results = leases.items()
102 results.extend(hosts)
85 return results103 return results
86104
=== modified file 'src/provisioningserver/dhcp/tests/test_leases_parser.py'
--- src/provisioningserver/dhcp/tests/test_leases_parser.py 2016-05-11 18:50:52 +0000
+++ src/provisioningserver/dhcp/tests/test_leases_parser.py 2017-02-07 17:35:36 +0000
@@ -41,26 +41,32 @@
4141
42class Lease(object):42class Lease(object):
4343
44 def __init__(self, lease_or_host, host, fixed_address, hardware, ends):44 def __init__(
45 self, lease_or_host, host, fixed_address, hardware, starts, ends,
46 binding_state=None):
45 self.lease_or_host = lease_or_host47 self.lease_or_host = lease_or_host
46 self.host = host48 self.host = host
47 self.hardware = hardware49 self.hardware = hardware
50 self.starts = starts
48 setattr(self, 'fixed-address', fixed_address)51 setattr(self, 'fixed-address', fixed_address)
49 self.ends = ends52 self.ends = ends
53 if binding_state is not None:
54 setattr(self, 'binding state', binding_state)
5055
51 def __iter__(self):56 def __iter__(self):
52 return iter(self.__dict__.keys())57 return iter(self.__dict__.keys())
5358
5459
55def fake_parsed_lease(ip=None, mac=None, ends=None,60def fake_parsed_lease(ip=None, mac=None, starts=None, ends=None,
56 entry_type='lease'):61 entry_type='lease', state=None):
57 """Fake a lease as produced by the parser."""62 """Fake a lease as produced by the parser."""
58 if ip is None:63 if ip is None:
59 ip = factory.make_ipv4_address()64 ip = factory.make_ipv4_address()
60 if mac is None:65 if mac is None:
61 mac = factory.make_mac_address()66 mac = factory.make_mac_address()
62 Hardware = namedtuple('Hardware', ['mac'])67 Hardware = namedtuple('Hardware', ['mac'])
63 lease = Lease(entry_type, ip, ip, Hardware(mac), ends)68 lease = Lease(
69 entry_type, ip, ip, Hardware(mac), starts, ends, binding_state=state)
64 return lease70 return lease
6571
6672
@@ -312,7 +318,13 @@
312 """ % params))318 """ % params))
313 self.assertEqual([(params['ip'], params['mac'])], leases)319 self.assertEqual([(params['ip'], params['mac'])], leases)
314320
315 def test_parse_leases_takes_all_leases_for_address(self):321 def test_parse_leases_takes_current_lease_for_address(self):
322 # Note: a previous version of this test case checked that two leases
323 # can exist at the same time for a single IP address. This has been
324 # removed in order to prevent stale entries from polluting MAAS's
325 # discoveries with old information. To support bonds, static host
326 # mappings will still allow more than one MAC address to be assigned to
327 # the same IP address.
316 params = {328 params = {
317 'ip': factory.make_ipv4_address(),329 'ip': factory.make_ipv4_address(),
318 'old_owner': factory.make_mac_address(),330 'old_owner': factory.make_mac_address(),
@@ -321,15 +333,35 @@
321 leases = self.parse(dedent("""\333 leases = self.parse(dedent("""\
322 lease %(ip)s {334 lease %(ip)s {
323 hardware ethernet %(old_owner)s;335 hardware ethernet %(old_owner)s;
336 ends 0 1990/01/01 00:00:00;
324 }337 }
325 lease %(ip)s {338 lease %(ip)s {
326 hardware ethernet %(new_owner)s;339 hardware ethernet %(new_owner)s;
327 }340 }
328 """ % params))341 """ % params))
329 self.assertEqual(342 self.assertEqual(
330 [(params['ip'], params['old_owner']),343 [(params['ip'], params['new_owner'])], leases)
331 (params['ip'], params['new_owner'])],344
332 leases)345 def test_multiple_host_declarations_are_reported(self):
346 params = {
347 'ip': factory.make_ipv4_address(),
348 'bondmac1': factory.make_mac_address(),
349 'bondmac2': factory.make_mac_address(),
350 }
351 leases = self.parse(dedent("""\
352 host %(bondmac1)s {
353 hardware ethernet %(bondmac1)s;
354 fixed-address %(ip)s;
355 }
356 host %(bondmac2)s {
357 hardware ethernet %(bondmac2)s;
358 fixed-address %(ip)s;
359 }
360 """ % params))
361 self.assertEqual([
362 (params['ip'], params['bondmac1']),
363 (params['ip'], params['bondmac2'])
364 ], leases)
333365
334 def test_parse_leases_recognizes_host_deleted_statement_as_rubout(self):366 def test_parse_leases_recognizes_host_deleted_statement_as_rubout(self):
335 params = {367 params = {
@@ -362,6 +394,39 @@
362394
363class TestLeasesParserFast(MAASTestCase):395class TestLeasesParserFast(MAASTestCase):
364396
397 def test_handles_dash_separator_for_host_mapping(self):
398 ip = factory.make_ipv4_address()
399 mac = factory.make_mac_address()
400 mac_dash = mac.replace(":", "-")
401 leases = leases_parser_fast.parse_leases(dedent("""\
402 host %s {
403 hardware ethernet %s;
404 fixed-address %s;
405 }
406 """ % (mac_dash, mac, ip)))
407 self.assertEqual([(ip, mac)], leases)
408
409 def test_multiple_host_declarations_are_reported(self):
410 params = {
411 'ip': factory.make_ipv4_address(),
412 'bondmac1': factory.make_mac_address(),
413 'bondmac2': factory.make_mac_address(),
414 }
415 leases = leases_parser_fast.parse_leases(dedent("""\
416 host %(bondmac1)s {
417 hardware ethernet %(bondmac1)s;
418 fixed-address %(ip)s;
419 }
420 host %(bondmac2)s {
421 hardware ethernet %(bondmac2)s;
422 fixed-address %(ip)s;
423 }
424 """ % params))
425 self.assertEqual([
426 (params['ip'], params['bondmac1']),
427 (params['ip'], params['bondmac2'])
428 ], leases)
429
365 def test_expired_lease_does_not_shadow_earlier_host_stanza(self):430 def test_expired_lease_does_not_shadow_earlier_host_stanza(self):
366 params = {431 params = {
367 'ip': factory.make_ipv4_address(),432 'ip': factory.make_ipv4_address(),
@@ -402,33 +467,55 @@
402 }467 }
403 """ % params))468 """ % params))
404 # The lease hasn't expired, so shadows the earlier host stanza.469 # The lease hasn't expired, so shadows the earlier host stanza.
405 self.assertEqual(470 # (But since the host mapping has not been removed, it takes precedence
406 [(params["ip"], params["mac1"]), (params["ip"], params["mac2"])],471 # by coming later in the list, since that should better describe the
407 leases)472 # intent of the MAAS administrator.)
408
409 def test_host_stanza_shadows_earlier_active_lease(self):
410 params = {
411 'ip': factory.make_ipv4_address(),
412 'mac1': factory.make_mac_address(),
413 'mac2': factory.make_mac_address(),
414 }
415 leases = leases_parser_fast.parse_leases(dedent("""\
416 lease %(ip)s {
417 starts 5 2010/01/01 00:00:01;
418 hardware ethernet %(mac2)s;
419 }
420 host %(ip)s {
421 dynamic;
422 hardware ethernet %(mac1)s;
423 fixed-address %(ip)s;
424 }
425 """ % params))
426 # The lease hasn't expired, but the host entry is later, so it
427 # shadows the earlier lease stanza.
428 self.assertEqual(473 self.assertEqual(
429 [(params["ip"], params["mac2"]), (params["ip"], params["mac1"])],474 [(params["ip"], params["mac2"]), (params["ip"], params["mac1"])],
430 leases)475 leases)
431476
477 def test_host_stanza_replaces_earlier_active_lease(self):
478 params = {
479 'ip': factory.make_ipv4_address(),
480 'mac1': factory.make_mac_address(),
481 'mac2': factory.make_mac_address(),
482 }
483 leases = leases_parser_fast.parse_leases(dedent("""\
484 lease %(ip)s {
485 starts 5 2010/01/01 00:00:01;
486 hardware ethernet %(mac2)s;
487 }
488 host %(ip)s {
489 dynamic;
490 hardware ethernet %(mac1)s;
491 fixed-address %(ip)s;
492 }
493 """ % params))
494 # The lease hasn't expired, but the host entry is later. So the host
495 # mapping takes precedence.
496 self.assertEqual(
497 [(params["ip"], params["mac1"])], leases)
498
499 def test_released_lease_with_no_end_time_is_released(self):
500 params = {
501 'ip': factory.make_ipv4_address(),
502 'mac1': factory.make_mac_address(),
503 'mac2': factory.make_mac_address(),
504 }
505 leases = leases_parser_fast.parse_leases(dedent("""\
506 lease %(ip)s {
507 starts 5 2010/01/01 00:00:01;
508 hardware ethernet %(mac2)s;
509 }
510 lease %(ip)s {
511 dynamic;
512 binding state free;
513 starts 0 1990/01/01 00:00:00;
514 }
515 """ % params))
516 # The lease was added and then removed, so we expect a no-op.
517 self.assertEqual([], leases)
518
432519
433class TestLeasesParserFunctions(MAASTestCase):520class TestLeasesParserFunctions(MAASTestCase):
434521
@@ -440,6 +527,14 @@
440 hour=03, minute=04, second=05),527 hour=03, minute=04, second=05),
441 get_expiry_date(lease))528 get_expiry_date(lease))
442529
530 def test_get_expiry_date_uses_start_date_for_free_lease(self):
531 lease = fake_parsed_lease(starts='0 2011/01/02 03:04:05', state='free')
532 self.assertEqual(
533 datetime(
534 year=2011, month=01, day=02,
535 hour=03, minute=04, second=05),
536 get_expiry_date(lease))
537
443 def test_get_expiry_date_returns_None_for_never(self):538 def test_get_expiry_date_returns_None_for_never(self):
444 self.assertIsNone(539 self.assertIsNone(
445 get_expiry_date(fake_parsed_lease(ends='never')))540 get_expiry_date(fake_parsed_lease(ends='never')))
@@ -492,7 +587,7 @@
492 ]587 ]
493 self.assertEqual([(ip, new_owner)], gather_leases(leases))588 self.assertEqual([(ip, new_owner)], gather_leases(leases))
494589
495 def test_gather_leases_ignores_ordering(self):590 def test_ordering_is_important_to_gather_leases(self):
496 earlier = '1 2001/01/01 00:00:00'591 earlier = '1 2001/01/01 00:00:00'
497 ip = factory.make_ipv4_address()592 ip = factory.make_ipv4_address()
498 old_owner = factory.make_mac_address()593 old_owner = factory.make_mac_address()
@@ -501,7 +596,7 @@
501 fake_parsed_lease(ip=ip, mac=new_owner),596 fake_parsed_lease(ip=ip, mac=new_owner),
502 fake_parsed_lease(ip=ip, mac=old_owner, ends=earlier),597 fake_parsed_lease(ip=ip, mac=old_owner, ends=earlier),
503 ]598 ]
504 self.assertEqual([(ip, new_owner)], gather_leases(leases))599 self.assertEqual([], gather_leases(leases))
505600
506 def test_gather_leases_ignores_host_declarations(self):601 def test_gather_leases_ignores_host_declarations(self):
507 self.assertEqual([], gather_leases([fake_parsed_host()]))602 self.assertEqual([], gather_leases([fake_parsed_host()]))

Subscribers

People subscribed via source and target branches