Merge lp:~mpontillo/maas/device-api-1566109 into lp:~maas-committers/maas/trunk

Proposed by Mike Pontillo
Status: Merged
Approved by: Mike Pontillo
Approved revision: no longer in the source branch.
Merged at revision: 4937
Proposed branch: lp:~mpontillo/maas/device-api-1566109
Merge into: lp:~maas-committers/maas/trunk
Diff against target: 160 lines (+92/-4)
4 files modified
src/maasserver/api/tests/test_devices.py (+18/-0)
src/maasserver/forms.py (+8/-0)
src/maasserver/tests/test_forms_device.py (+64/-2)
src/maasserver/utils/forms.py (+2/-2)
To merge this branch: bzr merge lp:~mpontillo/maas/device-api-1566109
Reviewer Review Type Date Requested Status
Mike Pontillo (community) Approve
Gavin Panella (community) Approve
Review via email: mp+291580@code.launchpad.net

Commit message

Ensure QueryDict is always used with WithMACAddressesMixin.

 * This is the likely fix for bug #1566109.

To post a comment you must log in.
Revision history for this message
Gavin Panella (allenap) wrote :

There's a problem in WithMACAddressesMixin that illustrates one of my
pet hates.

I want to pass a MultiValueDict into the form's constructor, so that
self.data has a getlist() method. This is used by WithMACAddressesMixin.
However, Django's BaseForm has the following code:

  class BaseForm(object):
      def __init__(self, data=None, ...):
          ...
          self.data = data or {}
          ...

so when I pass an *empty* MultiValueDict in — which is an ancestor of
QueryDict, the container that Django uses to convey request arguments to
handlers, and thus a perfectly reasonable thing to pass into a form —
it's replaced with a plain old Python dict, and subsequent attempts to
use getlist() on it break with AttributeError.

All because someone was too f***ing lazy test against None:

    self.data = {} if data is None else data

My blood boils. Implicit boolean-ness is a curse.

Revision history for this message
Gavin Panella (allenap) wrote :

I've got the tests passing in lp:~allenap/maas/device-api-1566109 (diff:
https://paste.ubuntu.com/15789749/). This was far harder than it ought
to have been. It has upgraded my dislike for Django's form machinery
from "moderate" to "strong".

WithMACAddressesMixin expects self.data to be a QueryDict. I haven't
double checked, but it seems that not even a MultiValueDict is enough.

Django's BaseForm works on the assumption that self.data is a plain
dict, so we're already in dodgy territory, because plain dicts and
QueryDicts do not quack quite the same way.

But maybe Django's all okay with this, because its MultipleHiddenInput
and SelectMultiple widgets have the following code:

    def value_from_datadict(self, data, files, name):
        if isinstance(data, (MultiValueDict, MergeDict)):
            return data.getlist(name)
        return data.get(name, None)

I don't know. It's a mess.

Revision history for this message
Gavin Panella (allenap) wrote :

Looks good. A few small things to tidy up, nothing more. Also, describe the fix in the commit message; the bug number is automatically added to the commit message by the lander so there's no need to mention it again (or at all).

review: Approve
Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (1.0 MiB)

The attempt to merge lp:~mpontillo/maas/device-api-1566109 into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main Sources [865 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Sources [7,757 kB]
Get:7 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,182 kB]
Get:8 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,541 kB]
Fetched 17.6 MB in 3s (5,067 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-8).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-8).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P4-8).
firefox is already the newest version (45.0.2+build1-0ubuntu1).
freeipmi-...

Revision history for this message
Mike Pontillo (mpontillo) wrote :

Could not reproduce random failure in maasserver.rpc.tests.test_regionservice.

Trying again to land.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'src/maasserver/api/tests/test_devices.py'
--- src/maasserver/api/tests/test_devices.py 2016-04-14 15:25:36 +0000
+++ src/maasserver/api/tests/test_devices.py 2016-04-19 16:54:49 +0000
@@ -133,6 +133,24 @@
133 self.assertEquals(domain, device.domain)133 self.assertEquals(domain, device.domain)
134 self.assertEqual(device.node_type, NODE_TYPE.DEVICE)134 self.assertEqual(device.node_type, NODE_TYPE.DEVICE)
135135
136 def test_POST_without_macs_raises_appropriate_error(self):
137 hostname = factory.make_name('host')
138 domain = factory.make_Domain()
139 response = self.client.post(
140 reverse('devices_handler'),
141 {
142 'hostname': hostname,
143 'domain': domain.name,
144 })
145 self.assertEqual(
146 http.client.BAD_REQUEST, response.status_code, response.content)
147
148 def test_empty_POST_raises_appropriate_error(self):
149 response = self.client.post(
150 reverse('devices_handler'), {})
151 self.assertEqual(
152 http.client.BAD_REQUEST, response.status_code, response.content)
153
136 def test_POST_returns_limited_fields(self):154 def test_POST_returns_limited_fields(self):
137 response = self.client.post(155 response = self.client.post(
138 reverse('devices_handler'),156 reverse('devices_handler'),
139157
=== modified file 'src/maasserver/forms.py'
--- src/maasserver/forms.py 2016-04-15 22:14:33 +0000
+++ src/maasserver/forms.py 2016-04-19 16:54:49 +0000
@@ -55,6 +55,7 @@
55 Form,55 Form,
56 MultipleChoiceField,56 MultipleChoiceField,
57)57)
58from django.http import QueryDict
58from django.utils.safestring import mark_safe59from django.utils.safestring import mark_safe
59from lxml import etree60from lxml import etree
60from maasserver.api.utils import get_overridden_query_dict61from maasserver.api.utils import get_overridden_query_dict
@@ -128,6 +129,7 @@
128from maasserver.utils.converters import machine_readable_bytes129from maasserver.utils.converters import machine_readable_bytes
129from maasserver.utils.forms import (130from maasserver.utils.forms import (
130 compose_invalid_choice_text,131 compose_invalid_choice_text,
132 get_QueryDict,
131 set_form_error,133 set_form_error,
132)134)
133from maasserver.utils.orm import (135from maasserver.utils.orm import (
@@ -1055,6 +1057,12 @@
10551057
1056 def __init__(self, *args, **kwargs):1058 def __init__(self, *args, **kwargs):
1057 super(WithMACAddressesMixin, self).__init__(*args, **kwargs)1059 super(WithMACAddressesMixin, self).__init__(*args, **kwargs)
1060 # This is a workaround for a Django bug in which self.data (which is
1061 # supposed to be a QueryDict) ends up being a normal Python dict.
1062 # This class requires a QueryDict (which it seems like Django should
1063 # enforce, for consistency).
1064 if not isinstance(self.data, QueryDict):
1065 self.data = get_QueryDict(self.data)
1058 self.set_up_mac_addresses_field()1066 self.set_up_mac_addresses_field()
10591067
1060 def set_up_mac_addresses_field(self):1068 def set_up_mac_addresses_field(self):
10611069
=== modified file 'src/maasserver/tests/test_forms_device.py'
--- src/maasserver/tests/test_forms_device.py 2016-03-28 13:54:47 +0000
+++ src/maasserver/tests/test_forms_device.py 2016-04-19 16:54:49 +0000
@@ -5,10 +5,27 @@
55
6__all__ = []6__all__ = []
77
8from maasserver.forms import DeviceForm8from django.http import HttpRequest
9from django.http.request import QueryDict
10from maasserver.forms import (
11 DeviceForm,
12 DeviceWithMACsForm,
13)
14from maasserver.models import (
15 Device,
16 Interface,
17)
9from maasserver.testing.factory import factory18from maasserver.testing.factory import factory
10from maasserver.testing.testcase import MAASServerTestCase19from maasserver.testing.testcase import MAASServerTestCase
11from maasserver.utils.orm import reload_object20from maasserver.utils.forms import get_QueryDict
21from maasserver.utils.orm import (
22 get_one,
23 reload_object,
24)
25from testtools.matchers import (
26 Contains,
27 Equals,
28)
1229
1330
14class TestDeviceForm(MAASServerTestCase):31class TestDeviceForm(MAASServerTestCase):
@@ -39,3 +56,48 @@
39 reload_object(parent)56 reload_object(parent)
4057
41 self.assertEqual(parent, device.parent)58 self.assertEqual(parent, device.parent)
59
60
61class TestDeviceWithMACsForm(MAASServerTestCase):
62
63 def make_request(self):
64 """Return a :class:`HttpRequest` with the given parameters."""
65 request = HttpRequest()
66 request.user = factory.make_User()
67 return request
68
69 def test_contains_mac_addresses_field_and_converts_non_querydict(self):
70 form = DeviceWithMACsForm(data={})
71 self.assertThat(form.fields, Contains('mac_addresses'))
72 self.assertIsInstance(form.data, QueryDict)
73
74 def test_creates_device_with_mac(self):
75 hostname = factory.make_name("device")
76 mac = factory.make_mac_address()
77 form = DeviceWithMACsForm(data=get_QueryDict({
78 "hostname": hostname,
79 "mac_addresses": mac
80 }), request=self.make_request())
81 self.assertTrue(form.is_valid(), dict(form.errors))
82 form.save()
83 device = get_one(Device.objects.filter(hostname=hostname))
84 self.assertThat(device.hostname, Equals(hostname))
85 iface = get_one(Interface.objects.filter(mac_address=mac))
86 self.assertThat(iface.node, Equals(device))
87
88 def test_creates_device_with_macs(self):
89 hostname = factory.make_name("device")
90 mac1 = factory.make_mac_address()
91 mac2 = factory.make_mac_address()
92 form = DeviceWithMACsForm(data=get_QueryDict({
93 "hostname": hostname,
94 "mac_addresses": [mac1, mac2]
95 }), request=self.make_request())
96 self.assertTrue(form.is_valid(), dict(form.errors))
97 form.save()
98 device = get_one(Device.objects.filter(hostname=hostname))
99 self.assertThat(device.hostname, Equals(hostname))
100 iface = get_one(Interface.objects.filter(mac_address=mac1))
101 self.assertThat(iface.node, Equals(device))
102 iface = get_one(Interface.objects.filter(mac_address=mac2))
103 self.assertThat(iface.node, Equals(device))
42104
=== modified file 'src/maasserver/utils/forms.py'
--- src/maasserver/utils/forms.py 2015-12-01 18:12:59 +0000
+++ src/maasserver/utils/forms.py 2016-04-19 16:54:49 +0000
@@ -32,9 +32,9 @@
32 )32 )
3333
3434
35def get_QueryDict(params):35def get_QueryDict(params, mutable=True):
36 """Convert `params` to a `QueryDict`."""36 """Convert `params` to a `QueryDict`."""
37 query_dict = QueryDict('', mutable=True)37 query_dict = QueryDict('', mutable=mutable)
38 for k, v in params.items():38 for k, v in params.items():
39 if isinstance(v, list):39 if isinstance(v, list):
40 query_dict.setlist(k, v)40 query_dict.setlist(k, v)