Merge lp:~allenap/maas/storage-node-special-filesystems-websocket-api into lp:~maas-committers/maas/trunk

[Gavin Panella]

Add special filesystems to nodes transmitted via the WebSocket API.

The deprecated LoopingCall.deferred attribute is also no longer used by the listener.

[Blake Rouse (blake-rouse)]

Looks good.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~allenap/maas/storage-node-special-filesystems-websocket-api into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease [116 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main Sources [1,106 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Sources [7,508 kB]
Get:7 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,433 kB]
Get:8 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,243 kB]
Fetched 17.4 MB in 3s (4,748 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-1ubuntu1).
archdetect-deb is already the newest version (1.114ubuntu3).
authbind is already the newest version (2.1.1+nmu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-4).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-4).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu2).
dh-apport is already the newest version (2.20-0ubuntu3).
dh-systemd is already the newest version (1.29ubuntu1).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.d...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~allenap/maas/storage-node-special-filesystems-websocket-api into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease [116 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main Sources [1,106 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Sources [7,506 kB]
Get:7 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,433 kB]
Get:8 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main Translation-en [730 kB]
Get:9 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,241 kB]
Get:10 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Translation-en [4,185 kB]
Fetched 22.3 MB in 4s (5,541 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-1ubuntu1).
archdetect-deb is already the newest version (1.114ubuntu3).
authbind is already the newest version (2.1.1+nmu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-4).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-4).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu2).
dh-apport is a...

=== modified file 'src/maasserver/api/tests/test_enlistment.py'
--- src/maasserver/api/tests/test_enlistment.py	2016-03-09 12:21:54 +0000
+++ src/maasserver/api/tests/test_enlistment.py	2016-03-21 16:05:02 +0000
@@ -433,10 +433,7 @@
                 'status_name',
                 'status_message',
                 'status_action',
-                # XXX: GavinPanella 2016-03-09 bug=1555022:
-                # special_filesystems SHOULD be amongst the fields rendered,
-                # but is not, and neither Lee nor I could figure out why.
-                # 'special_filesystems',
+                'special_filesystems',
             ],
             list(parsed_result))
 

=== modified file 'src/maasserver/listener.py'
--- src/maasserver/listener.py	2016-02-17 16:12:30 +0000
+++ src/maasserver/listener.py	2016-03-21 16:05:02 +0000
@@ -92,6 +92,7 @@
         self.connectionFileno = None
         self.notifications = set()
         self.notifier = task.LoopingCall(self.handleNotifies)
+        self.notifierDone = None
         self.connecting = None
         self.disconnecting = None
         self.registeredChannels = False
@@ -387,14 +388,13 @@
     def runHandleNotify(self, delay=0, clock=reactor):
         """Defer later the `handleNotify`."""
         if not self.notifier.running:
-            self.notifier.start(delay, now=False)
+            self.notifierDone = self.notifier.start(delay, now=False)
 
     def cancelHandleNotify(self):
         """Cancel the deferred `handleNotify` call."""
         if self.notifier.running:
-            done = self.notifier.deferred
             self.notifier.stop()
-            return done
+            return self.notifierDone
         else:
             return succeed(None)
 

=== modified file 'src/maasserver/models/filesystem.py'
--- src/maasserver/models/filesystem.py	2016-03-09 12:21:54 +0000
+++ src/maasserver/models/filesystem.py	2016-03-21 16:05:02 +0000
@@ -100,7 +100,8 @@
         BlockDevice, unique=False, null=True, blank=True)
 
     node = ForeignKey(
-        "Node", unique=False, null=True, blank=True)
+        "Node", unique=False, null=True, blank=True,
+        related_name="special_filesystems")
 
     # XXX: For CharField, why allow null *and* blank? Would
     # CharField(null=False, blank=True, default="") not work better?

=== modified file 'src/maasserver/websockets/handlers/node.py'
--- src/maasserver/websockets/handlers/node.py	2016-03-10 17:18:02 +0000
+++ src/maasserver/websockets/handlers/node.py	2016-03-21 16:05:02 +0000
@@ -7,6 +7,7 @@
     "NodeHandler",
 ]
 
+from itertools import chain
 import logging
 from operator import itemgetter
 
@@ -45,13 +46,14 @@
     return (
         queryset
         .select_related('boot_interface', 'owner', 'zone', 'domain')
-        .prefetch_related(
-            'interface_set__ip_addresses__subnet__vlan__fabric')
+        .prefetch_related('blockdevice_set__physicalblockdevice')
+        .prefetch_related('blockdevice_set__virtualblockdevice')
         .prefetch_related('interface_set__ip_addresses__subnet__space')
+        .prefetch_related('interface_set__ip_addresses__subnet__vlan__fabric')
         .prefetch_related('interface_set__vlan__fabric')
+        .prefetch_related('special_filesystems')
         .prefetch_related('tags')
-        .prefetch_related('blockdevice_set__physicalblockdevice')
-        .prefetch_related('blockdevice_set__virtualblockdevice'))
+    )
 
 
 class NodeHandler(TimestampedModelHandler):
@@ -151,23 +153,23 @@
             data["on_network"] = obj.on_network()
 
             # Storage
-            data["disks"] = [
-                self.dehydrate_blockdevice(blockdevice, obj)
-                for blockdevice in blockdevices
-            ]
-            data["disks"] = data["disks"] + [
-                self.dehydrate_volume_group(volume_group)
-                for volume_group in VolumeGroup.objects.filter_by_node(obj)
-            ] + [
-                self.dehydrate_cache_set(cache_set)
-                for cache_set in CacheSet.objects.get_cache_sets_for_node(obj)
-            ]
-            data["disks"] = sorted(data["disks"], key=itemgetter("name"))
+            data["disks"] = sorted(chain(
+                (self.dehydrate_blockdevice(blockdevice, obj)
+                 for blockdevice in blockdevices),
+                (self.dehydrate_volume_group(volume_group) for volume_group
+                 in VolumeGroup.objects.filter_by_node(obj)),
+                (self.dehydrate_cache_set(cache_set) for cache_set
+                 in CacheSet.objects.get_cache_sets_for_node(obj)),
+            ), key=itemgetter("name"))
             data["supported_filesystems"] = [
                 {'key': key, 'ui': ui}
                 for key, ui in FILESYSTEM_FORMAT_TYPE_CHOICES
             ]
             data["storage_layout_issues"] = obj.storage_layout_issues()
+            data["special_filesystems"] = [
+                self.dehydrate_filesystem(filesystem)
+                for filesystem in obj.special_filesystems.order_by("id")
+            ]
 
             # Events
             data["events"] = self.dehydrate_events(obj)

=== modified file 'src/maasserver/websockets/handlers/tests/test_device.py'
--- src/maasserver/websockets/handlers/tests/test_device.py	2016-02-26 18:39:26 +0000
+++ src/maasserver/websockets/handlers/tests/test_device.py	2016-03-21 16:05:02 +0000
@@ -209,7 +209,7 @@
         # number means regiond has to do more work slowing down its process
         # and slowing down the client waiting for the response.
         self.assertEqual(
-            query_10_count, 11,
+            query_10_count, 12,
             "Number of queries has changed; make sure this is expected.")
         self.assertEqual(
             query_10_count, query_20_count,

=== modified file 'src/maasserver/websockets/handlers/tests/test_machine.py'
--- src/maasserver/websockets/handlers/tests/test_machine.py	2016-03-17 16:37:11 +0000
+++ src/maasserver/websockets/handlers/tests/test_machine.py	2016-03-21 16:05:02 +0000
@@ -175,6 +175,10 @@
             "physical_disk_count": node.physicalblockdevice_set.count(),
             "disks": disks,
             "storage_layout_issues": node.storage_layout_issues(),
+            "special_filesystems": [
+                handler.dehydrate_filesystem(filesystem)
+                for filesystem in node.special_filesystems.order_by("id")
+            ],
             "supported_filesystems": [
                 {'key': key, 'ui': ui}
                 for key, ui in FILESYSTEM_FORMAT_TYPE_CHOICES],
@@ -828,6 +832,23 @@
             self.dehydrate_node(node, handler, include_summary=True),
             handler.get({"system_id": node.system_id}))
 
+    def test_get_includes_special_filesystems(self):
+        owner = factory.make_User()
+        handler = MachineHandler(owner, {})
+        machine = factory.make_Node(owner=owner)
+        filesystems = [
+            factory.make_Filesystem(node=machine),
+            factory.make_Filesystem(node=machine),
+        ]
+        self.assertThat(
+            handler.get({"system_id": machine.system_id}),
+            ContainsDict({
+                "special_filesystems": Equals([
+                    handler.dehydrate_filesystem(filesystem)
+                    for filesystem in filesystems
+                ]),
+            }))
+
     def test_list(self):
         user = factory.make_User()
         handler = MachineHandler(user, {})
@@ -863,7 +884,7 @@
         # number means regiond has to do more work slowing down its process
         # and slowing down the client waiting for the response.
         self.assertEqual(
-            query_10_count, 9,
+            query_10_count, 10,
             "Number of queries has changed; make sure this is expected.")
         self.assertEqual(
             query_10_count, query_20_count,