Merge lp:~jtv/maas/use-compose-curtin-network-preseed into lp:~maas-committers/maas/trunk

[Jeroen T. Vermeulen]

Configure networking on a node (when using Curtin) by asking the operating system driver on the cluster controller to generate configuration from scratch.

For now, that means “on Ubuntu generate /etc/network/interfaces and /etc/udev/rules.d/70-persistent-net.rules, and otherwise, do nothing.” This replaces the code that uploaded a script to the node if the OS was Ubuntu, and executed it to amend networking config.

[Graham Binns (gmb)]

Approved, with comments; something needs to change w.r.t exception handling, but there are a couple of ways you can do it, so I'll leave it up to you.

review: Approve

[Jeroen T. Vermeulen (jtv)]

Removed the pointless exception transformations. Thanks for figuring that out.

[Jeroen T. Vermeulen (jtv)]

Yay! It comes through the CI and it can deploy nodes. Still working to get a node without IPv4 set up though.

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~jtv/maas/use-compose-curtin-network-preseed into lp:maas failed. Below is the output from the failed tests.

Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [59.7 kB]
Get:5 http://security.ubuntu.com trusty-security/main Sources [46.3 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Get:6 http://security.ubuntu.com trusty-security/universe Sources [10.8 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Get:7 http://security.ubuntu.com trusty-security/main amd64 Packages [146 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:8 http://security.ubuntu.com trusty-security/universe amd64 Packages [49.0 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [125 kB]
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [86.2 kB]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [337 kB]
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [209 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Fetched 1,130 kB in 0s (1,886 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted python-oops-wsgi p...

[Jeroen T. Vermeulen (jtv)]

Lint from review change: imports became unused!

=== modified file 'src/maasserver/networking_preseed.py'
--- src/maasserver/networking_preseed.py	2014-10-02 04:14:09 +0000
+++ src/maasserver/networking_preseed.py	2014-10-02 13:17:35 +0000
@@ -25,10 +25,14 @@
 
 __metaclass__ = type
 __all__ = [
+    'compose_curtin_network_preseed_for',
     'generate_networking_config',
     ]
 
+import json
+
 from lxml import etree
+from maasserver.clusterrpc.osystems import compose_curtin_network_preseed
 from maasserver.dns.zonegenerator import get_dns_server_address
 from maasserver.exceptions import UnresolvableHost
 from maasserver.models.nodeprobeddetails import get_probed_details
@@ -317,3 +321,19 @@
     if len(macs) == 0:
         macs = node.macaddress_set.all()
     return [normalise_mac(unicode(mac.mac_address)) for mac in macs]
+
+
+def compose_curtin_network_preseed_for(node):
+    """Compose OS-dependent preseed for configuring networking on `node`.
+
+    :param node: A `Node`.
+    :return: A list of preseed strings.
+    """
+    config = {
+        'interfaces': extract_network_interfaces(node),
+        'auto_interfaces': find_macs_for_automatic_interfaces(node),
+        'ips_mapping': map_static_ips(node),
+        'gateways_mapping': map_gateways(node),
+        }
+    preseed = compose_curtin_network_preseed(node, config)
+    return [json.dumps(item) for item in preseed]

=== modified file 'src/maasserver/preseed.py'
--- src/maasserver/preseed.py	2014-10-01 16:50:53 +0000
+++ src/maasserver/preseed.py	2014-10-02 13:17:35 +0000
@@ -52,6 +52,7 @@
     Config,
     DHCPLease,
     )
+from maasserver.networking_preseed import compose_curtin_network_preseed_for
 from maasserver.node_status import COMMISSIONING_LIKE_STATUSES
 from maasserver.server_address import get_maas_facing_server_host
 from maasserver.third_party_drivers import get_third_party_driver
@@ -135,6 +136,7 @@
     return [yaml.safe_dump(reporter)]
 
 
+# This function is being replaced with compose_curtin_network_preseed_for.
 def compose_curtin_network_preseed(node):
     """Return a list of curtin preseeds for configuring a node's networking.
 
@@ -205,7 +207,7 @@
     installer_url = get_curtin_installer_url(node)
     main_config = get_curtin_config(node)
     reporter_config = compose_curtin_maas_reporter(node)
-    network_config = compose_curtin_network_preseed(node)
+    network_config = compose_curtin_network_preseed_for(node)
     return pack_install(
         configs=[main_config] + reporter_config + network_config,
         args=[installer_url])

=== modified file 'src/maasserver/rpc/testing/mixins.py'
--- src/maasserver/rpc/testing/mixins.py	2014-09-10 16:20:31 +0000
+++ src/maasserver/rpc/testing/mixins.py	2014-10-02 13:17:35 +0000
@@ -22,14 +22,19 @@
     RunningEventLoopFixture,
     )
 from maasserver.testing.factory import factory
-from provisioningserver.rpc.cluster import GetPreseedData
+from provisioningserver.rpc.cluster import (
+    ComposeCurtinNetworkPreseed,
+    GetPreseedData,
+    )
+from provisioningserver.rpc.testing import always_succeed_with
 
 
 class PreseedRPCMixin:
     """Set-up a live RPC environment for testing.
 
     It creates a cluster connected by RPC that responds to the
-    `GetPreseedData` call with a simple `NotImplementedError`.
+    `GetPreseedData` call with a simple `NotImplementedError`.  The
+    `ComposeCurtinNetworkPreseed` call returns an empty list.
 
     Tests that mix this in can use their ``rpc_cluster`` attribute (an
     instance of :py:class:`MockLiveRegionToClusterRPCFixture`) to control the
@@ -48,6 +53,8 @@
         # GetPreseedData call with a simple NotImplementedError.
         self.rpc_nodegroup = factory.make_NodeGroup()
         self.rpc_cluster = self.rpc_fixture.makeCluster(
-            self.rpc_nodegroup, GetPreseedData)
+            self.rpc_nodegroup, GetPreseedData, ComposeCurtinNetworkPreseed)
         self.rpc_cluster.GetPreseedData.side_effect = (
             NotImplementedError())
+        self.rpc_cluster.ComposeCurtinNetworkPreseed.side_effect = (
+            always_succeed_with({'data': []}))

=== modified file 'src/maasserver/tests/test_networking_preseed.py'
--- src/maasserver/tests/test_networking_preseed.py	2014-10-02 04:12:31 +0000
+++ src/maasserver/tests/test_networking_preseed.py	2014-10-02 13:17:35 +0000
@@ -15,6 +15,7 @@
 __all__ = [
     ]
 
+import json
 from random import randint
 
 from maasserver import networking_preseed
@@ -26,6 +27,7 @@
 from maasserver.exceptions import UnresolvableHost
 from maasserver.networking_preseed import (
     add_ip_to_mapping,
+    compose_curtin_network_preseed_for,
     extract_mac_string,
     extract_network_interfaces,
     find_macs_for_automatic_interfaces,
@@ -43,6 +45,7 @@
 from maasserver.testing.factory import factory
 from maasserver.testing.testcase import MAASServerTestCase
 from maastesting.matchers import MockCalledOnceWith
+from netaddr import IPAddress
 from testtools.matchers import HasLength
 
 
@@ -754,3 +757,51 @@
         self.assertItemsEqual(
             [normalise_mac(unicode(mac1)), normalise_mac(unicode(mac2))],
             find_macs_for_automatic_interfaces(node))
+
+
+class TestComposeCurtinNetworkPreseedFor(MAASServerTestCase):
+
+    def test__composes_config(self):
+        fake = self.patch_autospec(
+            networking_preseed, 'compose_curtin_network_preseed')
+        fake.return_value = []
+        node = factory.make_Node()
+        node.nodegroup.accept()
+        network = factory.make_ipv4_network(slash=24)
+        router = factory.pick_ip_in_network(network)
+        static_low = unicode(IPAddress(network.first + 1))
+        static_high = unicode(IPAddress(network.first + 2))
+        dyn_low = unicode(IPAddress(network.first + 3))
+        dyn_high = unicode(IPAddress(network.first + 4))
+        interface = factory.make_NodeGroupInterface(
+            node.nodegroup, network=network, router_ip=router,
+            static_ip_range_low=static_low, static_ip_range_high=static_high,
+            ip_range_low=dyn_low, ip_range_high=dyn_high,
+            management=NODEGROUPINTERFACE_MANAGEMENT.DHCP)
+        mac = factory.make_MACAddress(node=node, cluster_interface=interface)
+        extract_interfaces = self.patch_autospec(
+            networking_preseed, 'extract_network_interfaces')
+        extract_interfaces.return_value = [mac.mac_address]
+
+        compose_curtin_network_preseed_for(node)
+
+        expected_config = {
+            'interfaces': [mac.mac_address],
+            'auto_interfaces': [mac.mac_address],
+            'ips_mapping': {},
+            'gateways_mapping': {mac.mac_address: [router]},
+            }
+        self.assertThat(fake, MockCalledOnceWith(node, expected_config))
+
+    def test__returns_preseeds_as_list_of_text(self):
+        fake = self.patch_autospec(
+            networking_preseed, 'compose_curtin_network_preseed')
+        preseed_data = {factory.make_name('key'): factory.make_name('value')}
+        fake.return_value = [preseed_data]
+        node = factory.make_Node()
+
+        preseed = compose_curtin_network_preseed_for(node)
+
+        self.assertIsInstance(preseed, list)
+        [data] = preseed
+        self.assertEqual(preseed_data, json.loads(data))

=== modified file 'src/maasserver/tests/test_preseed.py'
--- src/maasserver/tests/test_preseed.py	2014-09-25 19:38:57 +0000
+++ src/maasserver/tests/test_preseed.py	2014-10-02 13:17:35 +0000
@@ -74,6 +74,7 @@
 from maasserver.utils import absolute_reverse
 from maastesting.matchers import MockCalledOnceWith
 from metadataserver.models import NodeKey
+from mock import ANY
 from provisioningserver.rpc.exceptions import NoConnectionsAvailable
 from provisioningserver.utils import locate_config
 from provisioningserver.utils.enum import map_enum
@@ -885,8 +886,14 @@
             node.nodegroup, management=NODEGROUPINTERFACE_MANAGEMENT.DHCP)
         arch, subarch = node.architecture.split('/')
         self.configure_get_boot_images_for_node(node, 'xinstall')
+
         user_data = get_curtin_userdata(node)
-        # Just check that the user data looks good.
+
+        self.expectThat(
+            self.rpc_cluster.ComposeCurtinNetworkPreseed,
+            MockCalledOnceWith(
+                ANY, osystem=node.get_osystem(), config=ANY,
+                disable_ipv4=node.disable_ipv4))
         self.assertIn("PREFIX='curtin'", user_data)