Merge lp:~mpontillo/maas/ip-allocation-from-incorrect-range--bug-1677507--2.1 into lp:maas/2.1

[Mike Pontillo]

Don't allow allocation of least-recently-used discoveries if they occur in a reserved range.

 * Merges revision 5887 from trunk to fix bug #1677507.

[Mike Pontillo (mpontillo)]

Rejecting temporarily to remove test case for feature that didn't exist in MAAS 2.1.

[Mike Pontillo (mpontillo)]

Self-approve backport.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/ip-allocation-from-incorrect-range--bug-1677507--2.1 into lp:maas/2.1 failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 0s (780 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
distro-info is already the newest version (0.14build1).
distro-info set to manually installed.
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-jquery set to manually installed.
make is already the newest version (4.1-6).
python-simplejson is already the newest version (3.8.1-1ubuntu2).
python-simplejson set to manually installed.
python-yaml is already the newest version (3.11-3build1).
python3-apt is already the newest version (1.1.0~beta1build1).
python3-jinja2 is already the newest version (2.8-1).
python3-oauthlib is already the newest version (1.0.3-1).
python3-requests is already the newest version (2.9.1-3).
python3-yaml is already the newest version (3.11-3build1).
ubuntu-cloudimage-keyring is already the newest version (2013.11.11).
bash is already the newest version (4.3-14ubuntu1.1).
bzr is already the newest version (2.7.0-2ubuntu3).
curl is already the newest version (7.47.0-1ubuntu2.2).
dnsutils is alr...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/ip-allocation-from-incorrect-range--bug-1677507--2.1 into lp:maas/2.1 failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Get:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 0s (684 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10....

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/ip-allocation-from-incorrect-range--bug-1677507--2.1 into lp:maas/2.1 failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 0s (697 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10....

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/ip-allocation-from-incorrect-range--bug-1677507--2.1 into lp:maas/2.1 failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Fetched 306 kB in 0s (695 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10....

=== modified file 'docs/changelog.rst'
--- docs/changelog.rst	2017-03-23 12:20:26 +0000
+++ docs/changelog.rst	2017-03-30 17:25:45 +0000
@@ -2,6 +2,15 @@
 Changelog
 =========
 
+2.1.6
+=====
+
+Bugs fixed in this release
+--------------------------
+
+LP: ##1677507   Fix potential IP allocation from incorrect range.
+
+
 2.1.5
 =====
 

=== modified file 'src/maasserver/models/subnet.py'
--- src/maasserver/models/subnet.py	2017-02-17 05:34:56 +0000
+++ src/maasserver/models/subnet.py	2017-03-30 17:25:45 +0000
@@ -593,8 +593,17 @@
         """
         # Circular imports.
         from maasserver.models import Discovery
-        return Discovery.objects.filter(
-            subnet=self).by_unknown_ip().order_by('last_seen').first()
+        # Note: for the purposes of this function, being in part of a "used"
+        # range (such as a router IP address or reserved range) makes it
+        # "known". So we need to avoid those here in order to avoid stepping
+        # on network infrastructure, reserved ranges, etc.
+        unused = self.get_ipranges_not_in_use(ignore_discovered_ips=True)
+        least_recent_neighbours = Discovery.objects.filter(
+            subnet=self).by_unknown_ip().order_by('last_seen')
+        for neighbor in least_recent_neighbours:
+            if neighbor.ip in unused:
+                return neighbor
+        return None
 
     def get_iprange_usage(self, with_neighbours=False) -> MAASIPSet:
         """Returns both the reserved and unreserved IP ranges in this Subnet.

=== modified file 'src/maasserver/models/tests/test_subnet.py'
--- src/maasserver/models/tests/test_subnet.py	2017-02-17 05:34:56 +0000
+++ src/maasserver/models/tests/test_subnet.py	2017-03-30 17:25:45 +0000
@@ -919,6 +919,23 @@
         discovery = subnet.get_least_recently_seen_unknown_neighbour()
         self.assertThat(discovery.ip, Equals("10.0.0.2"))
 
+    def test__returns_least_recently_seen_neighbour_excludes_in_use(self):
+        # Note: 10.0.0.0/30 --> 10.0.0.1 and 10.0.0.0.2 are usable.
+        subnet = factory.make_Subnet(
+            cidr="10.0.0.0/30", gateway_ip=None, dns_servers=None)
+        rackif = factory.make_Interface(vlan=subnet.vlan)
+        now = datetime.now()
+        yesterday = now - timedelta(days=1)
+        factory.make_Discovery(
+            ip="10.0.0.1", interface=rackif, updated=now)
+        factory.make_Discovery(
+            ip="10.0.0.2", interface=rackif, updated=yesterday)
+        factory.make_IPRange(
+            subnet, start_ip="10.0.0.2", end_ip="10.0.0.2",
+            type=IPRANGE_TYPE.RESERVED)
+        discovery = subnet.get_least_recently_seen_unknown_neighbour()
+        self.assertThat(discovery.ip, Equals("10.0.0.1"))
+
     def test__returns_none_if_no_neighbours(self):
         # Note: 10.0.0.0/30 --> 10.0.0.1 and 10.0.0.0.2 are usable.
         subnet = factory.make_Subnet(