Merge lp:~blake-rouse/maas/fix-xenial-poweroff into lp:~maas-committers/maas/trunk

[Blake Rouse]

Fix enlistment to work with Xenial. Fix commissioning on Xenial to ignore lxcbr0. Fix websocket protocol to work for sendError and sendNotify.

[Mike Pontillo (mpontillo)]

 Elegant. Looks good, assuming you also tested on trusty.

review: Approve

[Blake Rouse (blake-rouse)]

Yes this was tested with trusty. Disk erasing works on Trusty as well, who knows why Xenial does not work. I can't figure it out.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~blake-rouse/maas/fix-xenial-poweroff into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease [227 kB]
Hit:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:5 http://ppa.launchpad.net/maas-maintainers/experimental3/ubuntu xenial InRelease
Get:6 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main Sources [1,121 kB]
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,462 kB]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,078 kB]
Fetched 9,888 kB in 3s (2,590 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages were automatically installed and are no longer required:
  libc-ares-dev libc-ares2 libisl13 libv8-3.14-dev libv8-3.14.5
  linux-headers-4.3.0-1 linux-headers-4.3.0-1-generic
  linux-image-4.2.0-18-generic linux-image-4.3.0-1-generic
  linux-image-extra-4.3.0-1-generic python-twisted-lore
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  bind9 bind9-host bind9utils dnsutils libbind9-90 libdns-export100 libdns100
  libirs-export91 libisc-export95 libisc95 libisccc90 libisccfg-export90
  libisccfg90 liblwres90
14 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,060 kB of archives.
After this operation, 4,096 B disk space will be freed.
Get:1 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 libisc-export95 amd64 1:9.9.5.dfsg-12.1 [120 kB]
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 libdns-export100 amd64 1:9.9.5.dfsg-12.1 [427 kB]
Get:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 libisccfg-export90 amd64 1:9.9.5.dfsg-12.1 [19.9 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 libirs-export91 amd64 1:9.9.5.dfsg-12.1 [17.2 kB]
Get:5 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 bind9-host amd64 1:9.9.5.dfsg-12.1 [46.2 kB]
Get:6 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 dnsutils amd64 1:9.9.5.dfsg-12.1 [96.2 kB]
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 bind9 amd64 1:9.9.5.dfsg-12.1 [287 kB]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 libisc95 amd64 1:9.9.5.dfsg-12.1 [147 kB]
Get:9 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 libisccc90 amd64 1:9.9.5.dfsg-12.1 [15.5 kB]
Get:10 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 libisccfg90 amd64 1:9.9.5.dfsg-12.1 [36.2 kB]
Get:11 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 liblwres90 amd64 1:9.9.5.dfsg-12.1 [32.5 kB]
Get:12 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubunt...

=== modified file 'contrib/preseeds_v2/enlist_userdata'
--- contrib/preseeds_v2/enlist_userdata	2015-11-12 13:45:09 +0000
+++ contrib/preseeds_v2/enlist_userdata	2015-12-17 20:47:32 +0000
@@ -25,6 +25,12 @@
          primary:  "http://ports.ubuntu.com/ubuntu-ports"
          security: "http://ports.ubuntu.com/ubuntu-ports"
 
+power_state:
+   delay: now
+   mode: poweroff
+   timeout: 1800
+   condition: test ! -e /tmp/block-poweroff
+
 misc_bucket:
  - &maas_enlist |
    ####  IPMI setup  ######
@@ -149,40 +155,12 @@
       echo "sleeping 60 seconds then poweroff"
       echo
       echo "login with '$user:$pass' to debug and disable poweroff"
-      echo 
+      echo
       cat /tmp/enlist.out
       echo =============================================
       sleep 60
       [ -e $bfile ] && exit 0
    fi
- - &write_poweroff_job |
-   cat >/etc/init/maas-poweroff.conf <<EOF
-   description "poweroff when maas task is done"
-   start on stopped cloud-final
-   console output
-   task
-   script
-     [ ! -e /tmp/block-poweroff ] || exit 0
-     poweroff
-   end script
-   EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   initctl reload-configuration
- - &write_systemd_poweroff_job |
-   cat >/lib/systemd/system/maas-poweroff.service <<EOF
-   [Unit]
-   Description=Poweroff when maas task is done
-   Wants=cloud-final.service
-   After=cloud-final.service
-   ConditionPathExists=!/tmp/block-poweroff
-
-   [Service]
-   ExecStart=/sbin/poweroff
-   EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   if [ -d /run/systemd/system ]; then
-       systemctl daemon-reload
-   fi
 
 
 
@@ -190,5 +168,3 @@
 output: {all: '| tee -a /var/log/cloud-init-output.log'}
 runcmd:
  - [ sh, -c, *maas_enlist ]
- - [ sh, -c, *write_poweroff_job ]
- - [ sh, -c, *write_systemd_poweroff_job ]
 
=== modified file 'etc/maas/templates/commissioning-user-data/user_data.template'
--- etc/maas/templates/commissioning-user-data/user_data.template	2015-12-16 23:51:21 +0000
+++ etc/maas/templates/commissioning-user-data/user_data.template	2015-12-17 20:47:32 +0000
@@ -110,36 +110,6 @@
    exit 1
 }
 
-write_upstart_poweroff_job() {
-   cat >/etc/init/maas-poweroff.conf <<EOF
-   description "poweroff when maas task is done"
-   start on stopped cloud-final
-   console output
-   task
-   script
-     [ ! -e /tmp/block-poweroff ] || exit 0
-     poweroff
-   end script
-EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   initctl reload-configuration
-}
-
-write_systemd_poweroff_job() {
-   cat >/lib/systemd/system/maas-poweroff.service <<EOF
-   [Unit]
-   Description=Poweroff when maas task is done
-   Wants=cloud-final.service
-   After=cloud-final.service
-   ConditionPathExists=!/tmp/block-poweroff
-
-   [Service]
-   ExecStart=/sbin/poweroff
-EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   systemctl daemon-reload
-}
-
 write_block_poweroff() {
    touch /tmp/block-poweroff
 }
@@ -149,12 +119,6 @@
    write_block_poweroff
    {{endif}}
 
-   if [ -d /run/systemd/system ]; then
-     write_systemd_poweroff_job
-   else
-     write_upstart_poweroff_job
-   fi
-
    # Install tools and load modules.
    aptget update
    aptget install python3-yaml python3-oauthlib freeipmi-tools openipmi ipmitool
 
=== modified file 'etc/maas/templates/commissioning-user-data/user_data_disk_erasing.template'
--- etc/maas/templates/commissioning-user-data/user_data_disk_erasing.template	2015-12-16 23:51:21 +0000
+++ etc/maas/templates/commissioning-user-data/user_data_disk_erasing.template	2015-12-17 20:47:32 +0000
@@ -73,36 +73,6 @@
    exit 1
 }
 
-write_upstart_poweroff_job() {
-   cat >/etc/init/maas-poweroff.conf <<EOF
-   description "poweroff when maas task is done"
-   start on stopped cloud-final
-   console output
-   task
-   script
-     [ ! -e /tmp/block-poweroff ] || exit 0
-     poweroff
-   end script
-EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   initctl reload-configuration
-}
-
-write_systemd_poweroff_job() {
-   cat >/lib/systemd/system/maas-poweroff.service <<EOF
-   [Unit]
-   Description=Poweroff when maas task is done
-   Wants=cloud-final.service
-   After=cloud-final.service
-   ConditionPathExists=!/tmp/block-poweroff
-
-   [Service]
-   ExecStart=/sbin/poweroff
-EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   systemctl daemon-reload
-}
-
 erase_disks() {
    for disk in `lsblk -d -n -oKNAME,TYPE,RO | grep '^\w*\s*disk\s*0' | awk {'print $1'}`
    do
@@ -113,12 +83,6 @@
 }
 
 main() {
-   if [ -d /run/systemd/system ]; then
-     write_systemd_poweroff_job
-   else
-     write_upstart_poweroff_job
-   fi
-
    # Install tools and load modules.
    aptget update
    aptget install python3-yaml python3-oauthlib
 
=== modified file 'etc/maas/templates/commissioning-user-data/user_data_poweroff.template'
--- etc/maas/templates/commissioning-user-data/user_data_poweroff.template	2015-12-16 23:51:21 +0000
+++ etc/maas/templates/commissioning-user-data/user_data_poweroff.template	2015-12-17 20:47:32 +0000
@@ -1,46 +1,9 @@
 #!/bin/sh
 
-#### script setup ######
-PATH="$BIN_D:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
-
-write_upstart_poweroff_job() {
-   cat >/etc/init/maas-poweroff.conf <<EOF
-   description "Power-off when MAAS task is done"
-   start on stopped cloud-final
-   console output
-   task
-   script
-     [ ! -e /tmp/block-poweroff ] || exit 0
-     /sbin/poweroff
-   end script
-EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   initctl reload-configuration
-}
-
-write_systemd_poweroff_job() {
-   cat >/lib/systemd/system/maas-poweroff.service <<EOF
-   [Unit]
-   Description=Poweroff when maas task is done
-   Wants=cloud-final.service
-   After=cloud-final.service
-   ConditionPathExists=!/tmp/block-poweroff
-
-   [Service]
-   ExecStart=/sbin/poweroff
-EOF
-   # reload required due to lack of inotify in overlayfs (LP: #882147)
-   systemctl daemon-reload
-}
-
 main() {
-   if [ -d /run/systemd/system ]; then
-     write_systemd_poweroff_job
-   else
-     write_upstart_poweroff_job
-   fi
-
    echo "Powering node off."
+
+   # Cloud-init will handle powering off the node. Nothing to do.
 }
 
 main
 
=== modified file 'src/maasserver/compose_preseed.py'
--- src/maasserver/compose_preseed.py	2015-12-01 18:12:59 +0000
+++ src/maasserver/compose_preseed.py	2015-12-17 20:47:32 +0000
@@ -7,10 +7,14 @@
     'compose_preseed',
     ]
 
+from datetime import timedelta
 from urllib.parse import urlencode
 
 from maasserver.clusterrpc.osystems import get_preseed_data
-from maasserver.enum import PRESEED_TYPE
+from maasserver.enum import (
+    NODE_STATUS,
+    PRESEED_TYPE,
+)
 from maasserver.models.config import Config
 from maasserver.server_address import get_maas_facing_server_host
 from maasserver.utils import absolute_reverse
@@ -139,8 +143,13 @@
     """Compose the preseed value for a Commissioning node."""
     apt_proxy = get_apt_proxy_for_node(node)
     metadata_url = absolute_reverse('metadata', base_url=base_url)
+    poweroff_timeout = timedelta(hours=1).total_seconds()  # 1 hour
+    if node.status == NODE_STATUS.DISK_ERASING:
+        poweroff_timeout = timedelta(days=7).total_seconds()  # 1 week
     return _compose_cloud_init_preseed(
-        node, token, metadata_url, base_url=base_url, apt_proxy=apt_proxy)
+        node, token, metadata_url, base_url=base_url, apt_proxy=apt_proxy,
+        poweroff=True, poweroff_timeout=int(poweroff_timeout),
+        poweroff_condition="test ! -e /tmp/block-poweroff")
 
 
 def compose_curtin_preseed(node, token, base_url=''):
@@ -152,7 +161,8 @@
 
 
 def _compose_cloud_init_preseed(
-        node, token, metadata_url, base_url, apt_proxy=None):
+        node, token, metadata_url, base_url, apt_proxy=None,
+        poweroff=False, poweroff_timeout=3600, poweroff_condition=None):
     cloud_config = {
         'datasource': {
             'MAAS': {
@@ -179,6 +189,14 @@
     cloud_config.update(get_system_info())
     if apt_proxy:
         cloud_config['apt_proxy'] = apt_proxy
+    if poweroff:
+        cloud_config['power_state'] = {
+            'delay': 'now',
+            'mode': 'poweroff',
+            'timeout': poweroff_timeout,
+        }
+        if poweroff_condition is not None:
+            cloud_config['power_state']['condition'] = poweroff_condition
     return "#cloud-config\n%s" % yaml.safe_dump(cloud_config)
 
 
 
=== modified file 'src/maasserver/tests/test_compose_preseed.py'
--- src/maasserver/tests/test_compose_preseed.py	2015-12-01 18:12:59 +0000
+++ src/maasserver/tests/test_compose_preseed.py	2015-12-17 20:47:32 +0000
@@ -132,6 +132,28 @@
             absolute_reverse('metadata-status', args=[node.system_id]),
             preseed['reporting']['maas']['endpoint'])
 
+    def test_compose_preseed_for_commissioning_includes_poweroff(self):
+        node = factory.make_Node(status=NODE_STATUS.COMMISSIONING)
+        preseed = yaml.safe_load(
+            compose_preseed(PRESEED_TYPE.COMMISSIONING, node))
+        self.assertEquals({
+            'delay': 'now',
+            'mode': 'poweroff',
+            'timeout': 3600,
+            'condition': 'test ! -e /tmp/block-poweroff',
+        }, preseed['power_state'])
+
+    def test_compose_preseed_for_disk_erasing_includes_poweroff(self):
+        node = factory.make_Node(status=NODE_STATUS.DISK_ERASING)
+        preseed = yaml.safe_load(
+            compose_preseed(PRESEED_TYPE.COMMISSIONING, node))
+        self.assertEquals({
+            'delay': 'now',
+            'mode': 'poweroff',
+            'timeout': 604800,
+            'condition': 'test ! -e /tmp/block-poweroff',
+        }, preseed['power_state'])
+
     def test_compose_preseed_includes_node_oauth_token(self):
         node = factory.make_Node(status=NODE_STATUS.READY)
         node.nodegroup.accept()
 
=== modified file 'src/maasserver/websockets/protocol.py'
--- src/maasserver/websockets/protocol.py	2015-12-15 22:44:22 +0000
+++ src/maasserver/websockets/protocol.py	2015-12-17 20:47:32 +0000
@@ -324,7 +324,8 @@
             "rtype": RESPONSE_TYPE.ERROR,
             "error": error,
             }
-        self.transport.write(json.dumps(error_msg).encode("utf-8"))
+        self.transport.write(
+            json.dumps(error_msg, default=self._json_encode).encode("ascii"))
         return None
 
     def sendNotify(self, name, action, data):
@@ -335,7 +336,8 @@
             "action": action,
             "data": data,
             }
-        self.transport.write(json.dumps(notify_msg).encode("utf-8"))
+        self.transport.write(
+            json.dumps(notify_msg, default=self._json_encode).encode("ascii"))
 
     def buildHandler(self, handler_class):
         """Return an initialised instance of `handler_class`."""
 
=== modified file 'src/metadataserver/models/commissioningscript.py'
--- src/metadataserver/models/commissioningscript.py	2015-12-16 18:07:33 +0000
+++ src/metadataserver/models/commissioningscript.py	2015-12-17 20:47:32 +0000
@@ -178,6 +178,9 @@
         # Ignore loopback interfaces.
         if link_mac is None:
             continue
+        # Ignore lxcbr0 which is created by default on Xenial+.
+        elif link['name'] == 'lxcbr0':
+            continue
         else:
             ifname = link['name']
             try:
 
=== modified file 'src/metadataserver/models/tests/ip_addr_results.txt'
--- src/metadataserver/models/tests/ip_addr_results.txt	2015-07-16 08:30:38 +0000
+++ src/metadataserver/models/tests/ip_addr_results.txt	2015-12-17 20:47:32 +0000
@@ -2,21 +2,27 @@
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
-    inet6 ::1/128 scope host 
+    inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
     link/ether 00:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
     inet 192.168.0.3/24 brd 192.168.0.255 scope global eth0
        valid_lft forever preferred_lft forever
-    inet6 fe80::3e97:efe:fe0e:56dc/64 scope link 
-       valid_lft forever preferred_lft forever     
-    inet6 2001:db8:a::123/64 scope link 
+    inet6 fe80::3e97:efe:fe0e:56dc/64 scope link
+       valid_lft forever preferred_lft forever
+    inet6 2001:db8:a::123/64 scope link
        valid_lft forever preferred_lft forever
 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DORMANT group default qlen 1000
     link/ether 00:00:00:00:00:02 brd ff:ff:ff:ff:ff:ff
     inet 172.17.42.1/16 scope global eth1
        valid_lft forever preferred_lft forever
-3: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DORMANT group default qlen 1000
+4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DORMANT group default qlen 1000
     link/ether 00:00:00:00:00:03 brd ff:ff:ff:ff:ff:ff
     inet 172.17.12.1/16 scope global eth2
        valid_lft forever preferred_lft forever
+5: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
+    link/ether 00:00:00:00:00:04 brd ff:ff:ff:ff:ff:ff
+    inet 10.0.3.1/24 scope global lxcbr0
+       valid_lft forever preferred_lft forever
+    inet6 fe80::e477:64ff:fec5:25a3/64 scope link
+       valid_lft forever preferred_lft forever
 
=== modified file 'src/metadataserver/user_data/tests/test_poweroff.py'
--- src/metadataserver/user_data/tests/test_poweroff.py	2015-12-02 23:30:15 +0000
+++ src/metadataserver/user_data/tests/test_poweroff.py	2015-12-17 20:47:32 +0000
@@ -43,5 +43,4 @@
         self.assertThat(
             base64.b64decode(user_data_script.get_payload()), ContainsAll({
                 b'Powering node off',
-                b'poweroff',
             }))