Merge lp:~andreserl/maas/lp1646955 into lp:~maas-committers/maas/trunk

[Andres Rodriguez]

Re-add deprecated variables to the curtin preseed context so that MAAS doesn't fail rendering on machine deployment, causing machines to fail to deploy.

[Mike Pontillo (mpontillo)]

Some minor nits below.

[Blake Rouse (blake-rouse)]

Also missing commit message.

review: Needs Fixing

[Blake Rouse (blake-rouse)]

Looks good.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~andreserl/maas/lp1646955 into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Get:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 0s (612 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libnss-wrapper libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10.4-0.1build2).
python-psycopg2 is...

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Andres Rodriguez (andreserl)]

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~andreserl/maas/lp1646955 into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 2s (124 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libnss-wrapper libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10.4-0.1build2).
python-psycopg2 is...

[Andres Rodriguez (andreserl)]

review: Approve

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Andres Rodriguez (andreserl)]

review: Approve

=== modified file 'src/maasserver/preseed.py'
--- src/maasserver/preseed.py	2016-12-08 09:10:54 +0000
+++ src/maasserver/preseed.py	2016-12-12 17:30:38 +0000
@@ -62,7 +62,10 @@
 from metadataserver.models import NodeKey
 from metadataserver.user_data.snippets import get_snippet_context
 from provisioningserver.drivers.osystem.ubuntu import UbuntuOS
-from provisioningserver.logger import get_maas_logger
+from provisioningserver.logger import (
+    get_maas_logger,
+    LegacyLogger,
+)
 from provisioningserver.rpc.exceptions import NoConnectionsAvailable
 from provisioningserver.utils import typed
 from provisioningserver.utils.url import compose_URL
@@ -72,6 +75,8 @@
 
 maaslog = get_maas_logger("preseed")
 
+log = LegacyLogger()
+
 GENERIC_FILENAME = 'generic'
 
 
@@ -417,9 +422,36 @@
         get_node_preseed_context(
             node, osystem, series, rack_controller=rack_controller))
     context.update(get_curtin_context(node, rack_controller=rack_controller))
+    deprecated_context_variables = [
+        'main_archive_hostname', 'main_archive_directory',
+        'ports_archive_hostname', 'ports_archive_directory',
+        'enable_http_proxy', 'http_proxy']
+    deprecated_config_variables = []
+    for var in deprecated_context_variables:
+        if var not in context:
+            deprecated_context_variables.remove(var)
+    context.update(get_node_deprecated_preseed_context())
     config = yaml.load(template.substitute(**context))
+    # Remove deprecated config from the curtin preseed.
     if 'power_state' in config:
         del config['power_state']
+        deprecated_config_variables.append('power_state')
+    if 'apt_proxy' in config:
+        deprecated_config_variables.append('apt_proxy')
+        del config['apt_proxy']
+    if 'apt_mirrors' in config:
+        deprecated_config_variables.append('apt_mirrors')
+        del config['apt_mirrors']
+    if deprecated_context_variables:
+        log.warn(
+            "WARNING: '%s' contains deprecated preseed "
+            "variables. Please remove - %s" % (
+                template.name, ",".join(deprecated_context_variables)))
+    if deprecated_config_variables:
+        log.warn(
+            "WARNING: '%s' contains deprecated preseed "
+            "configuration. Please remove - %s" % (
+                template.name, ",".join(deprecated_config_variables)))
     # Precise does not support cloud-init performing the reboot, so curtin
     # must have this statement.
     if node.distro_series == "precise":
@@ -721,6 +753,30 @@
     }
 
 
+def get_node_deprecated_preseed_context():
+    """Return the node-dependent context dictionary to be used to render
+    preseed template. This includes all the context variables that have
+    been deprecated, but allows for backwards compatibility for those
+    preseeds that still contain old variables.
+
+    :return: The context dictionary.
+    :rtype: dict.
+    """
+    main_archive_hostname, main_archive_directory = get_netloc_and_path(
+        PackageRepository.get_main_archive().url)
+    ports_archive_hostname, ports_archive_directory = get_netloc_and_path(
+        PackageRepository.get_ports_archive().url)
+
+    return {
+        'main_archive_hostname': main_archive_hostname,
+        'main_archive_directory': main_archive_directory,
+        'ports_archive_hostname': ports_archive_hostname,
+        'ports_archive_directory': ports_archive_directory,
+        'enable_http_proxy': Config.objects.get_config('enable_http_proxy'),
+        'http_proxy': Config.objects.get_config('http_proxy'),
+    }
+
+
 def render_enlistment_preseed(
         prefix, osystem='', release='', rack_controller=None):
     """Return the enlistment preseed.

=== modified file 'src/maasserver/tests/test_preseed.py'
--- src/maasserver/tests/test_preseed.py	2016-12-08 09:10:54 +0000
+++ src/maasserver/tests/test_preseed.py	2016-12-12 17:30:38 +0000
@@ -54,6 +54,7 @@
     get_enlist_preseed,
     get_enlist_userdata,
     get_netloc_and_path,
+    get_node_deprecated_preseed_context,
     get_node_preseed_context,
     get_preseed,
     get_preseed_context,
@@ -460,6 +461,23 @@
             context)
 
 
+class TestNodeDeprecatedPreseedContext(
+        PreseedRPCMixin, BootImageHelperMixin, MAASTransactionServerTestCase):
+    """Test for `get_node_deprecated_preseed_context`."""
+
+    def test_get_node_deprecated_preseed_context_contains_keys(self):
+        node = factory.make_Node_with_Interface_on_Subnet(
+            primary_rack=self.rpc_rack_controller)
+        self.configure_get_boot_images_for_node(node, 'install')
+        context = get_node_deprecated_preseed_context()
+        self.assertItemsEqual(
+            ['main_archive_hostname', 'main_archive_directory',
+             'ports_archive_hostname', 'ports_archive_directory',
+             'enable_http_proxy', 'http_proxy'
+             ],
+            context)
+
+
 class TestNodePreseedContext(
         PreseedRPCMixin, BootImageHelperMixin, MAASTransactionServerTestCase):
     """Tests for `get_node_preseed_context`."""
@@ -880,6 +898,33 @@
         config = get_curtin_config(node)
         self.assertThat(config, Not(Contains('mode: reboot')))
 
+    def test_get_curtin_config_removes_apt_mirrors(self):
+        node = factory.make_Node_with_Interface_on_Subnet(
+            primary_rack=self.rpc_rack_controller)
+        self.configure_get_boot_images_for_node(node, 'xinstall')
+        apt_mirrors_template = dedent("""\
+        apt_mirrors:
+          ubuntu_archive:
+          ubuntu_security:
+        """)
+        self.patch(preseed_module, "get_preseed_template").return_value = (
+            factory.make_name("filename"), apt_mirrors_template)
+        config = get_curtin_config(node)
+        self.assertThat(config, Not(Contains('ubuntu_archive')))
+        self.assertThat(config, Not(Contains('ubuntu_security')))
+
+    def test_get_curtin_config_removes_apt_proxy(self):
+        node = factory.make_Node_with_Interface_on_Subnet(
+            primary_rack=self.rpc_rack_controller)
+        self.configure_get_boot_images_for_node(node, 'xinstall')
+        apt_proxy_template = dedent("""\
+        apt_proxy: http://127.0.0.1:8000/
+        """)
+        self.patch(preseed_module, "get_preseed_template").return_value = (
+            factory.make_name("filename"), apt_proxy_template)
+        config = get_curtin_config(node)
+        self.assertThat(config, Not(Contains('127.0.0.1')))
+
     def test_get_curtin_config_contains_reboot_for_precise(self):
         node = factory.make_Node_with_Interface_on_Subnet(
             primary_rack=self.rpc_rack_controller)