Merge lp:~jtv/maas/generate-node-networking-config-1 into lp:~maas-committers/maas/trunk

[Jeroen T. Vermeulen]

First part: generate a curtin network configuration preseed along the lines of the example that Scott provided.

We'll be able to send this into Curtin (how exactly I'm hoping to hear from Scott tonight) and it will configure the node's networking for us, once implementation in cloud-init is complete. According to Scott this is the only right way to configure the networking, so it's worth the extra work.

Jeroen

[Graham Binns (gmb)]

Some minor tweaks needed for clarity, but looks good! Thanks :)

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~jtv/maas/generate-node-networking-config-1 into lp:maas failed. Below is the output from the failed tests.

Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [59.7 kB]
Get:5 http://security.ubuntu.com trusty-security/main Sources [44.9 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Get:6 http://security.ubuntu.com trusty-security/universe Sources [10.8 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Get:7 http://security.ubuntu.com trusty-security/main amd64 Packages [144 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:8 http://security.ubuntu.com trusty-security/universe amd64 Packages [48.5 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [120 kB]
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [85.2 kB]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [322 kB]
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [205 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Fetched 1,101 kB in 0s (1,753 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted python-oops-wsgi py...

=== added file 'src/maasserver/networking_preseed.py'
--- src/maasserver/networking_preseed.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/networking_preseed.py	2014-09-24 02:49:10 +0000
@@ -0,0 +1,121 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Network configuration preseed code.
+
+This will eventually generate installer networking configuration like:
+
+   https://gist.github.com/jayofdoom/b035067523defec7fb53
+
+The installer running on a node will use this to set up the node's networking
+according to MAAS's specifications.
+"""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = [
+    'generate_networking_config',
+    ]
+
+from lxml import etree
+from maasserver.models.nodeprobeddetails import get_probed_details
+
+
+def extract_network_interface_data(element):
+    """Extract network interface name and MAC address from XML element.
+
+    :return: A tuple of network interface name and MAC address as found in
+        the XML.  If either is not found, it will be `None`.
+    """
+    interfaces = element.xpath("logicalname")
+    macs = element.xpath("serial")
+    if len(interfaces) == 0 or len(macs) == 0:
+        # Not enough data.
+        return None, None
+    assert len(interfaces) == 1
+    assert len(macs) == 1
+    return interfaces[0].text, macs[0].text
+
+
+def normalise_mac(mac):
+    """Return a MAC's normalised representation.
+
+    This doesn't actually parse all the different formats for writing MAC
+    addresses, but it does eliminate case differences.  In practice, any MAC
+    address this code is likely to will equal another version of itself after
+    normalisation, even if they were originally different spellings.
+    """
+    return mac.strip().lower()
+
+
+def extract_network_interfaces(node):
+    """Extract network interfaces from node's `lshw` output.
+
+    :param node: A `Node`.
+    :return: A list of tuples describing the network interfaces.
+        Each tuple consists of an interface name and a MAC address.
+    """
+    node_details = get_probed_details([node.system_id])
+    if node.system_id not in node_details:
+        return []
+    lshw_xml = node_details[node.system_id].get('lshw')
+    if lshw_xml is None:
+        return []
+    network_nodes = etree.fromstring(lshw_xml).xpath("//node[@id='network']")
+    interfaces = [
+        extract_network_interface_data(xml_node)
+        for xml_node in network_nodes
+        ]
+    return [
+        (interface, normalise_mac(mac))
+        for interface, mac in interfaces
+        if interface is not None and mac is not None
+        ]
+
+
+def generate_ethernet_link_entry(interface, mac):
+    """Generate the `links` list entry for the given ethernet interface.
+
+    :param interface: Network interface name, e.g. `eth0`.
+    :param mac: MAC address, e.g. `00:11:22:33:44:55`.
+    :return: A dict specifying the network interface, with keys
+        `id`, `type`, and `ethernet_mac_address` (and possibly more).
+    """
+    return {
+        'id': interface,
+        'type': 'ethernet',
+        'ethernet_mac_address': mac,
+        }
+
+
+def generate_networking_config(node):
+    """Generate a networking preseed for `node`.
+
+    :param node: A `Node`.
+    :return: A dict along the lines of the example in
+        https://gist.github.com/jayofdoom/b035067523defec7fb53 -- just
+        json-encode it to get a file in that format.
+    """
+    interfaces = extract_network_interfaces(node)
+    return {
+        'provider': "MAAS",
+        'network_info': {
+            'services': [
+                # List DNS servers here.
+                ],
+            'networks': [
+                # Write network specs here.
+                ],
+            'links': [
+                generate_ethernet_link_entry(interface, mac)
+                for interface, mac in interfaces
+                ],
+            },
+        }

=== added file 'src/maasserver/tests/test_networking_preseed.py'
--- src/maasserver/tests/test_networking_preseed.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/tests/test_networking_preseed.py	2014-09-24 02:49:10 +0000
@@ -0,0 +1,263 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for networking preseed code."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = [
+    ]
+
+from maasserver import networking_preseed
+from maasserver.networking_preseed import (
+    extract_network_interfaces,
+    generate_ethernet_link_entry,
+    generate_networking_config,
+    normalise_mac,
+    )
+from maasserver.testing.factory import factory
+from maasserver.testing.testcase import MAASServerTestCase
+from maastesting.matchers import MockCalledOnceWith
+
+
+def make_denormalised_mac():
+    return ' %s ' % factory.make_mac_address().upper()
+
+
+class TestExtractNetworkInterfaces(MAASServerTestCase):
+
+    def test__returns_nothing_if_no_lshw_output_found(self):
+        node = factory.make_Node()
+        self.assertEqual([], extract_network_interfaces(node))
+
+    def test__returns_nothing_if_no_network_description_found_in_lshw(self):
+        node = factory.make_Node()
+        lshw_output = """
+            <list xmlns:lldp="lldp" xmlns:lshw="lshw">
+              <lshw:list>
+              </lshw:list>
+            </list>
+            """
+        factory.make_NodeResult_for_commissioning(
+            node=node, name='00-maas-01-lshw.out', script_result=0,
+            data=lshw_output.encode('ascii'))
+        self.assertEqual([], extract_network_interfaces(node))
+
+    def test__extracts_interface_data(self):
+        node = factory.make_Node()
+        interface = factory.make_name('eth')
+        mac = factory.make_mac_address()
+        lshw_output = """
+            <node id="network" claimed="true" class="network">
+             <logicalname>%(interface)s</logicalname>
+             <serial>%(mac)s</serial>
+            </node>
+            """ % {'interface': interface, 'mac': mac}
+        factory.make_NodeResult_for_commissioning(
+            node=node, name='00-maas-01-lshw.out', script_result=0,
+            data=lshw_output.encode('ascii'))
+        self.assertEqual([(interface, mac)], extract_network_interfaces(node))
+
+    def test__finds_network_interface_on_motherboard(self):
+        node = factory.make_Node()
+        interface = factory.make_name('eth')
+        mac = factory.make_mac_address()
+        # Stripped-down version of real lshw output:
+        lshw_output = """
+            <!-- generated by lshw-B.02.16 -->
+            <list>
+            <node id="mynode" claimed="true" class="system" handle="DMI:0002">
+              <node id="core" claimed="true" class="bus" handle="DMI:0003">
+               <description>Motherboard</description>
+                <node id="pci" claimed="true" class="bridge" \
+                      handle="PCIBUS:0000:00">
+                 <description>Host bridge</description>
+                  <node id="network" claimed="true" class="network" \
+                      handle="PCI:0000:00:19.0">
+                   <description>Ethernet interface</description>
+                   <product>82566DM-2 Gigabit Network Connection</product>
+                   <vendor>Intel Corporation</vendor>
+                   <logicalname>%(interface)s</logicalname>
+                   <serial>%(mac)s</serial>
+                   <configuration>
+                    <setting id="ip" value="10.99.99.1" />
+                   </configuration>
+                  </node>
+                </node>
+              </node>
+            </node>
+            </list>
+            """ % {'interface': interface, 'mac': mac}
+        factory.make_NodeResult_for_commissioning(
+            node=node, name='00-maas-01-lshw.out', script_result=0,
+            data=lshw_output.encode('ascii'))
+        self.assertEqual([(interface, mac)], extract_network_interfaces(node))
+
+    def test__finds_network_interface_on_pci_bus(self):
+        node = factory.make_Node()
+        interface = factory.make_name('eth')
+        mac = factory.make_mac_address()
+        # Stripped-down version of real lshw output:
+        lshw_output = """
+            <!-- generated by lshw-B.02.16 -->
+            <list>
+            <node id="mynode" claimed="true" class="system" handle="DMI:0002">
+              <node id="core" claimed="true" class="bus" handle="DMI:0003">
+               <description>Motherboard</description>
+                <node id="pci" claimed="true" class="bridge" \
+                    handle="PCIBUS:0000:00">
+                 <description>Host bridge</description>
+                  <node id="pci:2" claimed="true" class="bridge" \
+                      handle="PCIBUS:0000:07">
+                   <description>PCI bridge</description>
+                    <node id="network" claimed="true" class="network" \
+                        handle="PCI:0000:07:04.0">
+                     <description>Ethernet interface</description>
+                     <logicalname>%(interface)s</logicalname>
+                     <serial>%(mac)s</serial>
+                     <configuration>
+                      <setting id="ip" value="192.168.1.114" />
+                     </configuration>
+                    </node>
+                  </node>
+                </node>
+              </node>
+            </node>
+            </list>
+            """ % {'interface': interface, 'mac': mac}
+        factory.make_NodeResult_for_commissioning(
+            node=node, name='00-maas-01-lshw.out', script_result=0,
+            data=lshw_output.encode('ascii'))
+        self.assertEqual([(interface, mac)], extract_network_interfaces(node))
+
+    def test__ignores_nodes_without_interface_name(self):
+        node = factory.make_Node()
+        mac = factory.make_mac_address()
+        lshw_output = """
+            <node id="network" claimed="true" class="network">
+             <serial>%s</serial>
+            </node>
+            """ % mac
+        factory.make_NodeResult_for_commissioning(
+            node=node, name='00-maas-01-lshw.out', script_result=0,
+            data=lshw_output.encode('ascii'))
+        self.assertEqual([], extract_network_interfaces(node))
+
+    def test__ignores_nodes_without_mac(self):
+        node = factory.make_Node()
+        interface = factory.make_name('eth')
+        lshw_output = """
+            <node id="network" claimed="true" class="network">
+             <logicalname>%s</logicalname>
+            </node>
+            """ % interface
+        factory.make_NodeResult_for_commissioning(
+            node=node, name='00-maas-01-lshw.out', script_result=0,
+            data=lshw_output.encode('ascii'))
+        self.assertEqual([], extract_network_interfaces(node))
+
+    def test__normalises_mac(self):
+        node = factory.make_Node()
+        interface = factory.make_name('eth')
+        mac = make_denormalised_mac()
+        self.assertNotEqual(normalise_mac(mac), mac)
+        lshw_output = """
+            <node id="network" claimed="true" class="network">
+             <logicalname>%(interface)s</logicalname>
+             <serial>%(mac)s</serial>
+            </node>
+            """ % {'interface': interface, 'mac': mac}
+        factory.make_NodeResult_for_commissioning(
+            node=node, name='00-maas-01-lshw.out', script_result=0,
+            data=lshw_output.encode('ascii'))
+        [entry] = extract_network_interfaces(node)
+        _, extracted_mac = entry
+        self.assertEqual(normalise_mac(mac), extracted_mac)
+
+
+class TestNormaliseMAC(MAASServerTestCase):
+
+    def test__normalises_case(self):
+        mac = factory.make_mac_address()
+        self.assertEqual(
+            normalise_mac(mac.lower()),
+            normalise_mac(mac.upper()))
+
+    def test__strips_whitespace(self):
+        mac = factory.make_mac_address()
+        self.assertEqual(
+            normalise_mac(mac),
+            normalise_mac(' %s ' % mac))
+
+    def test__is_idempotent(self):
+        mac = factory.make_mac_address()
+        self.assertEqual(
+            normalise_mac(mac),
+            normalise_mac(normalise_mac(mac)))
+
+
+class TestGenerateEthernetLinkEntry(MAASServerTestCase):
+
+    def test__generates_dict(self):
+        interface = factory.make_name('eth')
+        mac = factory.make_mac_address()
+        self.assertEqual(
+            {
+                'id': interface,
+                'type': 'ethernet',
+                'ethernet_mac_address': mac,
+            },
+            generate_ethernet_link_entry(interface, mac))
+
+
+class TestGenerateNetworkingConfig(MAASServerTestCase):
+
+    def patch_interfaces(self, interface_mac_pairs):
+        patch = self.patch_autospec(
+            networking_preseed, 'extract_network_interfaces')
+        patch.return_value = interface_mac_pairs
+        return patch
+
+    def test__returns_config_dict(self):
+        self.patch_interfaces([])
+        config = generate_networking_config(factory.make_Node())
+        self.assertIsInstance(config, dict)
+        self.assertEqual("MAAS", config['provider'])
+
+    def test__includes_links(self):
+        node = factory.make_Node()
+        interface = factory.make_name('eth')
+        mac = factory.make_mac_address()
+        patch = self.patch_interfaces([(interface, mac)])
+
+        config = generate_networking_config(node)
+
+        self.assertThat(patch, MockCalledOnceWith(node))
+        self.assertEqual(
+            [
+                {
+                    'id': interface,
+                    'type': 'ethernet',
+                    'ethernet_mac_address': mac,
+                },
+            ],
+            config['network_info']['links'])
+
+    def test__includes_networks(self):
+        # This section is not yet implemented, so expect an empty list.
+        self.patch_interfaces([])
+        config = generate_networking_config(factory.make_Node())
+        self.assertEqual([], config['network_info']['networks'])
+
+    def test__includes_dns_servers(self):
+        # This section is not yet implemented, so expect an empty list.
+        self.patch_interfaces([])
+        config = generate_networking_config(factory.make_Node())
+        self.assertEqual([], config['network_info']['services'])